[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:28:19 dist-test-slave-d59m krb5kdc[23437](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:28:19 dist-test-slave-d59m krb5kdc[23437](info): set up 2 sockets
Sep 28 06:28:19 dist-test-slave-d59m krb5kdc[23437](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 06:28:20.256865 23430 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Sep 28 06:28:20 dist-test-slave-d59m krb5kdc[23437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:28:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:28:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab.
Entry for principal kudu/127.22.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab.
I20260928 06:28:20.383584 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:39897
--webserver_interface=127.22.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab
--principal=kudu/127.22.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:44425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.225.190:39897
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:28:20.480914 23450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:28:20.481163 23450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:28:20.481206 23450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:28:20.482460 23450 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:28:20.482527 23450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:28:20.482553 23450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:28:20.482584 23450 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:28:20.482638 23450 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:28:20.484184 23450 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:44425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.225.190:39897
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab
--principal=kudu/127.22.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:39897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:28:20.484476 23450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:28:20.484776 23450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:28:20.493706 23455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:20.493698 23456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:20.493698 23458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:20.494110 23450 server_base.cc:1034] running on GCE node
I20260928 06:28:20.494369 23450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:28:20.494750 23450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:28:20.495911 23450 hybrid_clock.cc:648] HybridClock initialized: now 1790576900495893 us; error 87 us; skew 500 ppm
Sep 28 06:28:20 dist-test-slave-d59m krb5kdc[23437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576900, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:28:20.498730 23450 init.cc:377] Logged in from keytab as kudu/127.22.225.190@KRBTEST.COM (short username kudu)
I20260928 06:28:20.499294 23450 webserver.cc:460] Webserver started at http://127.22.225.190:39811/ using document root <none> and password file <none>
I20260928 06:28:20.499552 23450 fs_manager.cc:359] Metadata directory not provided
I20260928 06:28:20.499605 23450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:28:20.499723 23450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:28:20.501379 23450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/minicluster-data/master-0/data/instance:
uuid: "463ea797ce454a64b51efeaf6cdaf8f5"
format_stamp: "Formatted at 2026-09-28 06:28:20 on dist-test-slave-d59m"
server_key: "318b584924c957ac07a11c11d018e21a"
server_key_iv: "a0f26ca65e2dcab0203807d3e88e3a69"
server_key_version: "encryptionkey@0"
I20260928 06:28:20.502134 23450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/minicluster-data/master-0/wal/instance:
uuid: "463ea797ce454a64b51efeaf6cdaf8f5"
format_stamp: "Formatted at 2026-09-28 06:28:20 on dist-test-slave-d59m"
server_key: "318b584924c957ac07a11c11d018e21a"
server_key_iv: "a0f26ca65e2dcab0203807d3e88e3a69"
server_key_version: "encryptionkey@0"
I20260928 06:28:20.503782 23450 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:28:20.504791 23465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:20.505014 23450 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:28:20.505115 23450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/minicluster-data/master-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/minicluster-data/master-0/wal
uuid: "463ea797ce454a64b51efeaf6cdaf8f5"
format_stamp: "Formatted at 2026-09-28 06:28:20 on dist-test-slave-d59m"
server_key: "318b584924c957ac07a11c11d018e21a"
server_key_iv: "a0f26ca65e2dcab0203807d3e88e3a69"
server_key_version: "encryptionkey@0"
I20260928 06:28:20.505210 23450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:28:20.540268 23450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:28:20.543049 23450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:28:20.543211 23450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:28:20.548264 23450 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.190:39897
I20260928 06:28:20.548295 23517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.190:39897 every 8 connection(s)
I20260928 06:28:20.548745 23450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
I20260928 06:28:20.549324 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 23450
I20260928 06:28:20.549444 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/minicluster-data/master-0/wal/instance
I20260928 06:28:20.549873 23518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:20.550258 23430 external_mini_cluster.cc:1428] Setting key 1ba172630ee37d862d8b363bfa32c830
I20260928 06:28:20.554405 23518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 463ea797ce454a64b51efeaf6cdaf8f5: Bootstrap starting.
I20260928 06:28:20.555223 23518 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 463ea797ce454a64b51efeaf6cdaf8f5: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:20.555546 23518 log.cc:824] T 00000000000000000000000000000000 P 463ea797ce454a64b51efeaf6cdaf8f5: Log is configured to *not* fsync() on all Append() calls
I20260928 06:28:20.556412 23518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 463ea797ce454a64b51efeaf6cdaf8f5: No bootstrap required, opened a new log
I20260928 06:28:20.561501 23518 raft_consensus.cc:348] T 00000000000000000000000000000000 P 463ea797ce454a64b51efeaf6cdaf8f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "463ea797ce454a64b51efeaf6cdaf8f5" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 39897 } }
I20260928 06:28:20.561779 23518 raft_consensus.cc:374] T 00000000000000000000000000000000 P 463ea797ce454a64b51efeaf6cdaf8f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:20.561821 23518 raft_consensus.cc:729] T 00000000000000000000000000000000 P 463ea797ce454a64b51efeaf6cdaf8f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 463ea797ce454a64b51efeaf6cdaf8f5, State: Initialized, Role: FOLLOWER
I20260928 06:28:20.562000 23518 consensus_queue.cc:260] T 00000000000000000000000000000000 P 463ea797ce454a64b51efeaf6cdaf8f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "463ea797ce454a64b51efeaf6cdaf8f5" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 39897 } }
I20260928 06:28:20.562104 23518 raft_consensus.cc:388] T 00000000000000000000000000000000 P 463ea797ce454a64b51efeaf6cdaf8f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:28:20.562170 23518 raft_consensus.cc:482] T 00000000000000000000000000000000 P 463ea797ce454a64b51efeaf6cdaf8f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:28:20.562224 23518 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 463ea797ce454a64b51efeaf6cdaf8f5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:20.563000 23518 raft_consensus.cc:504] T 00000000000000000000000000000000 P 463ea797ce454a64b51efeaf6cdaf8f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "463ea797ce454a64b51efeaf6cdaf8f5" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 39897 } }
I20260928 06:28:20.563129 23518 leader_election.cc:302] T 00000000000000000000000000000000 P 463ea797ce454a64b51efeaf6cdaf8f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 463ea797ce454a64b51efeaf6cdaf8f5; no voters:
I20260928 06:28:20.563340 23518 leader_election.cc:288] T 00000000000000000000000000000000 P 463ea797ce454a64b51efeaf6cdaf8f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:28:20.563392 23523 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 463ea797ce454a64b51efeaf6cdaf8f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:28:20.563624 23523 raft_consensus.cc:686] T 00000000000000000000000000000000 P 463ea797ce454a64b51efeaf6cdaf8f5 [term 1 LEADER]: Becoming Leader. State: Replica: 463ea797ce454a64b51efeaf6cdaf8f5, State: Running, Role: LEADER
I20260928 06:28:20.563715 23518 sys_catalog.cc:564] T 00000000000000000000000000000000 P 463ea797ce454a64b51efeaf6cdaf8f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:28:20.563750 23523 consensus_queue.cc:237] T 00000000000000000000000000000000 P 463ea797ce454a64b51efeaf6cdaf8f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "463ea797ce454a64b51efeaf6cdaf8f5" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 39897 } }
I20260928 06:28:20.564117 23523 sys_catalog.cc:455] T 00000000000000000000000000000000 P 463ea797ce454a64b51efeaf6cdaf8f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "463ea797ce454a64b51efeaf6cdaf8f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "463ea797ce454a64b51efeaf6cdaf8f5" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 39897 } } }
I20260928 06:28:20.564235 23523 sys_catalog.cc:458] T 00000000000000000000000000000000 P 463ea797ce454a64b51efeaf6cdaf8f5 [sys.catalog]: This master's current role is: LEADER
I20260928 06:28:20.564399 23523 sys_catalog.cc:455] T 00000000000000000000000000000000 P 463ea797ce454a64b51efeaf6cdaf8f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 463ea797ce454a64b51efeaf6cdaf8f5. Latest consensus state: current_term: 1 leader_uuid: "463ea797ce454a64b51efeaf6cdaf8f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "463ea797ce454a64b51efeaf6cdaf8f5" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 39897 } } }
I20260928 06:28:20.564482 23523 sys_catalog.cc:458] T 00000000000000000000000000000000 P 463ea797ce454a64b51efeaf6cdaf8f5 [sys.catalog]: This master's current role is: LEADER
I20260928 06:28:20.564682 23529 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:28:20.565218 23529 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 06:28:20 dist-test-slave-d59m krb5kdc[23437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:28:20.567269 23529 catalog_manager.cc:1347] Generated new cluster ID: c1de463b316b41f183cd4149a06ab867
I20260928 06:28:20.568931 23529 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:28:20.571185 23520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:20.551353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59798 (local address 127.22.225.190:39897)
0928 06:28:20.551530 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:20.551539 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:20.551797 (+ 258us) server_negotiation.cc:407] Connection header received
0928 06:28:20.554313 (+ 2516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:20.554321 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:20.554383 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:20.554534 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:20.556821 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:20.559139 (+ 2318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:20.560918 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:20.561497 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:20.561614 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:20.566543 (+ 4929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:20.566547 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:20.566547 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:20.566561 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:20.568784 (+ 2223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:20.569286 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:20.569289 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:20.569290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:20.569427 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:20.569722 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:20.569723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:20.569723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:20.570014 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:20.570155 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:20.570856 (+ 701us) server_negotiation.cc:299] Negotiation successful
0928 06:28:20.570987 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":88,"threads_started":1}
I20260928 06:28:20.583068 23529 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:28:20.583801 23529 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:28:20.589500 23529 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 463ea797ce454a64b51efeaf6cdaf8f5: Generated new TSK 0
I20260928 06:28:20.589706 23529 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.225.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab.
Entry for principal kudu/127.22.225.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab.
I20260928 06:28:20.623832 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.129:0
--local_ip_for_outbound_sockets=127.22.225.129
--webserver_interface=127.22.225.129
--webserver_port=0
--tserver_master_addrs=127.22.225.190:39897
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab
--principal=kudu/127.22.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:44425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:28:20.742379 23544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:28:20.742661 23544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:28:20.742697 23544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:28:20.744439 23544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:28:20.744496 23544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:28:20.744551 23544 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.129
I20260928 06:28:20.746801 23544 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:44425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab
--principal=kudu/127.22.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.225.129
--webserver_port=0
--tserver_master_addrs=127.22.225.190:39897
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.129
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:28:20.747053 23544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:28:20.747360 23544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:28:20.753742 23550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:20.753738 23549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:20.753738 23552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:20.754148 23544 server_base.cc:1034] running on GCE node
I20260928 06:28:20.754346 23544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:28:20.754612 23544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:28:20.755776 23544 hybrid_clock.cc:648] HybridClock initialized: now 1790576900755760 us; error 33 us; skew 500 ppm
Sep 28 06:28:20 dist-test-slave-d59m krb5kdc[23437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576900, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:28:20.758373 23544 init.cc:377] Logged in from keytab as kudu/127.22.225.129@KRBTEST.COM (short username kudu)
I20260928 06:28:20.758867 23544 webserver.cc:460] Webserver started at http://127.22.225.129:36771/ using document root <none> and password file <none>
I20260928 06:28:20.759094 23544 fs_manager.cc:359] Metadata directory not provided
I20260928 06:28:20.759156 23544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:28:20.759296 23544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:28:20.760352 23544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/minicluster-data/ts-0/data/instance:
uuid: "ec2c8295e81b45d3b8c88eb6e82237f6"
format_stamp: "Formatted at 2026-09-28 06:28:20 on dist-test-slave-d59m"
server_key: "3aa9ffb7afd40d77d56104f641e185c1"
server_key_iv: "83a502e9d7a950469e4455dbe98d3e28"
server_key_version: "encryptionkey@0"
I20260928 06:28:20.760766 23544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/minicluster-data/ts-0/wal/instance:
uuid: "ec2c8295e81b45d3b8c88eb6e82237f6"
format_stamp: "Formatted at 2026-09-28 06:28:20 on dist-test-slave-d59m"
server_key: "3aa9ffb7afd40d77d56104f641e185c1"
server_key_iv: "83a502e9d7a950469e4455dbe98d3e28"
server_key_version: "encryptionkey@0"
I20260928 06:28:20.762135 23544 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:28:20.763058 23559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:20.763271 23544 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:28:20.763365 23544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/minicluster-data/ts-0/wal
uuid: "ec2c8295e81b45d3b8c88eb6e82237f6"
format_stamp: "Formatted at 2026-09-28 06:28:20 on dist-test-slave-d59m"
server_key: "3aa9ffb7afd40d77d56104f641e185c1"
server_key_iv: "83a502e9d7a950469e4455dbe98d3e28"
server_key_version: "encryptionkey@0"
I20260928 06:28:20.763445 23544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:28:20.778770 23544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:28:20.781154 23544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:28:20.781317 23544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:28:20.781556 23544 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:28:20.781875 23544 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:28:20.781934 23544 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:20.781976 23544 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:28:20.782006 23544 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:20.788090 23544 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.129:43311
I20260928 06:28:20.788152 23672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.129:43311 every 8 connection(s)
I20260928 06:28:20.788509 23544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
I20260928 06:28:20.790016 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 23544
I20260928 06:28:20.790117 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/minicluster-data/ts-0/wal/instance
I20260928 06:28:20.790269 23430 external_mini_cluster.cc:1428] Setting key 1083d59d85fe275dff4b2edc6bcbafeb
Sep 28 06:28:20 dist-test-slave-d59m krb5kdc[23437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576900, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.129@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.225.130@KRBTEST.COM; defaulting to no policy
I20260928 06:28:20.799388 23520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:20.789330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:46045 (local address 127.22.225.190:39897)
0928 06:28:20.789398 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:20.789400 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:20.789629 (+ 229us) server_negotiation.cc:407] Connection header received
0928 06:28:20.790337 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:20.790339 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:20.790352 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:20.790419 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:20.792016 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:20.792851 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:20.793910 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:20.794214 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:20.794259 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:20.796560 (+ 2301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:20.796561 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:20.796561 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:20.796569 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:20.798010 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:20.798426 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:20.798427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:20.798428 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:20.798493 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:20.798734 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:20.798735 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:20.798735 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:20.798846 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:20.798925 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:20.799215 (+ 290us) server_negotiation.cc:299] Negotiation successful
0928 06:28:20.799299 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 06:28:20.799598 23675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:20.789304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:39897 (local address 127.22.225.129:46045)
0928 06:28:20.789484 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:20.789533 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:20.790219 (+ 686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:20.790557 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:20.790561 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:20.791612 (+ 1051us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:20.791891 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:20.791899 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:20.792965 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:20.792966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:20.793806 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:20.793809 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:20.794343 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:20.794343 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:20.794448 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:20.795008 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:20.795019 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:20.796464 (+ 1445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:20.798108 (+ 1644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:20.798115 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:20.798117 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:20.798339 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:20.798558 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:20.798559 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:20.798559 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:20.798670 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:20.798910 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:20.798917 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:20.799073 (+ 156us) client_negotiation.cc:769] Sending connection context
0928 06:28:20.799183 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 06:28:20.799340 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":93,"threads_started":1}
I20260928 06:28:20.800077 23673 heartbeater.cc:347] Connected to a master server at 127.22.225.190:39897
I20260928 06:28:20.800184 23673 heartbeater.cc:464] Registering TS with master...
I20260928 06:28:20.800374 23673 heartbeater.cc:511] Master 127.22.225.190:39897 requested a full tablet report, sending...
I20260928 06:28:20.800937 23482 ts_manager.cc:194] Registered new tserver with Master: ec2c8295e81b45d3b8c88eb6e82237f6 (127.22.225.129:43311)
I20260928 06:28:20.801604 23482 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.225.129@KRBTEST.COM'} at 127.22.225.129:46045
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab.
Entry for principal kudu/127.22.225.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab.
I20260928 06:28:20.819079 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.130:0
--local_ip_for_outbound_sockets=127.22.225.130
--webserver_interface=127.22.225.130
--webserver_port=0
--tserver_master_addrs=127.22.225.190:39897
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab
--principal=kudu/127.22.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:44425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:28:20.906306 23678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:28:20.906508 23678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:28:20.906531 23678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:28:20.907665 23678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:28:20.907706 23678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:28:20.907732 23678 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.130
I20260928 06:28:20.909271 23678 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:44425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab
--principal=kudu/127.22.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.225.130
--webserver_port=0
--tserver_master_addrs=127.22.225.190:39897
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.130
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:28:20.909518 23678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:28:20.909739 23678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:28:20.914959 23683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:20.914991 23686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:20.914991 23684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:20.915423 23678 server_base.cc:1034] running on GCE node
I20260928 06:28:20.915639 23678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:28:20.915866 23678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:28:20.917021 23678 hybrid_clock.cc:648] HybridClock initialized: now 1790576900917005 us; error 32 us; skew 500 ppm
Sep 28 06:28:20 dist-test-slave-d59m krb5kdc[23437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576900, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:28:20.919708 23678 init.cc:377] Logged in from keytab as kudu/127.22.225.130@KRBTEST.COM (short username kudu)
I20260928 06:28:20.920317 23678 webserver.cc:460] Webserver started at http://127.22.225.130:34997/ using document root <none> and password file <none>
I20260928 06:28:20.920563 23678 fs_manager.cc:359] Metadata directory not provided
I20260928 06:28:20.920653 23678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:28:20.920789 23678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:28:20.922194 23678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/minicluster-data/ts-1/data/instance:
uuid: "9d4320d8e8a14e1e8753b7c407f4ae86"
format_stamp: "Formatted at 2026-09-28 06:28:20 on dist-test-slave-d59m"
server_key: "96d641d8603c42d9ab3b3f896c2ea3f4"
server_key_iv: "e5dd112704fbbe032d3f0b4db6d9ff7f"
server_key_version: "encryptionkey@0"
I20260928 06:28:20.922637 23678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/minicluster-data/ts-1/wal/instance:
uuid: "9d4320d8e8a14e1e8753b7c407f4ae86"
format_stamp: "Formatted at 2026-09-28 06:28:20 on dist-test-slave-d59m"
server_key: "96d641d8603c42d9ab3b3f896c2ea3f4"
server_key_iv: "e5dd112704fbbe032d3f0b4db6d9ff7f"
server_key_version: "encryptionkey@0"
I20260928 06:28:20.924297 23678 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:28:20.925202 23693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:20.925401 23678 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:28:20.925498 23678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/minicluster-data/ts-1/wal
uuid: "9d4320d8e8a14e1e8753b7c407f4ae86"
format_stamp: "Formatted at 2026-09-28 06:28:20 on dist-test-slave-d59m"
server_key: "96d641d8603c42d9ab3b3f896c2ea3f4"
server_key_iv: "e5dd112704fbbe032d3f0b4db6d9ff7f"
server_key_version: "encryptionkey@0"
I20260928 06:28:20.925567 23678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:28:20.938259 23678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:28:20.940647 23678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:28:20.940800 23678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:28:20.941056 23678 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:28:20.941403 23678 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:28:20.941445 23678 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:20.941473 23678 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:28:20.941494 23678 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:20.948469 23678 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.130:33703
I20260928 06:28:20.948531 23806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.130:33703 every 8 connection(s)
I20260928 06:28:20.948930 23678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
I20260928 06:28:20.954733 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 23678
I20260928 06:28:20.954842 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/minicluster-data/ts-1/wal/instance
I20260928 06:28:20.955013 23430 external_mini_cluster.cc:1428] Setting key bcfc6bf24a1668f3811115a3460489de
Sep 28 06:28:20 dist-test-slave-d59m krb5kdc[23437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576900, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.130@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.225.131@KRBTEST.COM; defaulting to no policy
I20260928 06:28:20.964094 23520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:20.949740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:60193 (local address 127.22.225.190:39897)
0928 06:28:20.949800 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:20.949802 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:20.950019 (+ 217us) server_negotiation.cc:407] Connection header received
0928 06:28:20.950561 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:20.950563 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:20.950576 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:20.950628 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:20.951582 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:20.952094 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:20.953204 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:20.953406 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:20.953478 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:20.960404 (+ 6926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:20.960406 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:20.960407 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:20.960417 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:20.962263 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:20.962782 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:20.962784 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:20.962785 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:20.962847 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:20.963151 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:20.963152 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:20.963153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:20.963317 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:20.963389 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:20.963840 (+ 451us) server_negotiation.cc:299] Negotiation successful
0928 06:28:20.963937 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 06:28:20.964260 23809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:20.949727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:39897 (local address 127.22.225.130:60193)
0928 06:28:20.949897 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:20.949935 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:20.950442 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:20.950702 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:20.950705 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:20.951285 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:20.951477 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:20.951482 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:20.952204 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:20.952205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:20.953115 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:20.953117 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:20.953496 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:20.953497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:20.953619 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:20.958396 (+ 4777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:20.958408 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:20.960276 (+ 1868us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:20.962363 (+ 2087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:20.962371 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:20.962374 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:20.962692 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:20.962930 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:20.962931 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:20.962931 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:20.963075 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:20.963440 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:20.963447 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:20.963650 (+ 203us) client_negotiation.cc:769] Sending connection context
0928 06:28:20.963792 (+ 142us) client_negotiation.cc:240] Negotiation successful
0928 06:28:20.963966 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":84,"threads_started":1}
I20260928 06:28:20.964846 23807 heartbeater.cc:347] Connected to a master server at 127.22.225.190:39897
I20260928 06:28:20.964936 23807 heartbeater.cc:464] Registering TS with master...
I20260928 06:28:20.965154 23807 heartbeater.cc:511] Master 127.22.225.190:39897 requested a full tablet report, sending...
I20260928 06:28:20.965557 23482 ts_manager.cc:194] Registered new tserver with Master: 9d4320d8e8a14e1e8753b7c407f4ae86 (127.22.225.130:33703)
I20260928 06:28:20.966131 23482 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.225.130@KRBTEST.COM'} at 127.22.225.130:60193
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab.
Entry for principal kudu/127.22.225.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab.
I20260928 06:28:20.981767 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.131:0
--local_ip_for_outbound_sockets=127.22.225.131
--webserver_interface=127.22.225.131
--webserver_port=0
--tserver_master_addrs=127.22.225.190:39897
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab
--principal=kudu/127.22.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:44425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:28:21.071679 23812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:28:21.071923 23812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:28:21.071972 23812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:28:21.073235 23812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:28:21.073295 23812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:28:21.073342 23812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.131
I20260928 06:28:21.074836 23812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:44425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab
--principal=kudu/127.22.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.225.131
--webserver_port=0
--tserver_master_addrs=127.22.225.190:39897
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.131
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:28:21.075109 23812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:28:21.075348 23812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:28:21.080446 23817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:21.080518 23818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:21.080622 23820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:21.080682 23812 server_base.cc:1034] running on GCE node
I20260928 06:28:21.080893 23812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:28:21.081184 23812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:28:21.082360 23812 hybrid_clock.cc:648] HybridClock initialized: now 1790576901082349 us; error 31 us; skew 500 ppm
Sep 28 06:28:21 dist-test-slave-d59m krb5kdc[23437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576901, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:28:21.085391 23812 init.cc:377] Logged in from keytab as kudu/127.22.225.131@KRBTEST.COM (short username kudu)
I20260928 06:28:21.085942 23812 webserver.cc:460] Webserver started at http://127.22.225.131:46875/ using document root <none> and password file <none>
I20260928 06:28:21.086180 23812 fs_manager.cc:359] Metadata directory not provided
I20260928 06:28:21.086246 23812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:28:21.086382 23812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:28:21.087563 23812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/minicluster-data/ts-2/data/instance:
uuid: "1ba0cc91ae81409dab5527500151bc8a"
format_stamp: "Formatted at 2026-09-28 06:28:21 on dist-test-slave-d59m"
server_key: "b4592ee6bea88e21de3e23d946417746"
server_key_iv: "5b6e1d84527c9a5ced4a81f70abf8a3a"
server_key_version: "encryptionkey@0"
I20260928 06:28:21.087996 23812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/minicluster-data/ts-2/wal/instance:
uuid: "1ba0cc91ae81409dab5527500151bc8a"
format_stamp: "Formatted at 2026-09-28 06:28:21 on dist-test-slave-d59m"
server_key: "b4592ee6bea88e21de3e23d946417746"
server_key_iv: "5b6e1d84527c9a5ced4a81f70abf8a3a"
server_key_version: "encryptionkey@0"
I20260928 06:28:21.089596 23812 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:28:21.090384 23827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:21.090559 23812 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:28:21.090654 23812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/minicluster-data/ts-2/wal
uuid: "1ba0cc91ae81409dab5527500151bc8a"
format_stamp: "Formatted at 2026-09-28 06:28:21 on dist-test-slave-d59m"
server_key: "b4592ee6bea88e21de3e23d946417746"
server_key_iv: "5b6e1d84527c9a5ced4a81f70abf8a3a"
server_key_version: "encryptionkey@0"
I20260928 06:28:21.090735 23812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:28:21.112026 23812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:28:21.114550 23812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:28:21.114722 23812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:28:21.114977 23812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:28:21.115307 23812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:28:21.115358 23812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:21.115403 23812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:28:21.115427 23812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:21.121552 23812 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.131:42447
I20260928 06:28:21.121609 23941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.131:42447 every 8 connection(s)
I20260928 06:28:21.121973 23812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
Sep 28 06:28:21 dist-test-slave-d59m krb5kdc[23437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576901, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.131@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:28:21.129290 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 23812
I20260928 06:28:21.129386 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576899143097-23430-0/minicluster-data/ts-2/wal/instance
I20260928 06:28:21.129540 23430 external_mini_cluster.cc:1428] Setting key 9e7304cc9482a40bf41409f36c6b5d6c
I20260928 06:28:21.132839 23520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:21.122751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:47567 (local address 127.22.225.190:39897)
0928 06:28:21.122822 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:21.122824 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:21.123103 (+ 279us) server_negotiation.cc:407] Connection header received
0928 06:28:21.123737 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:21.123738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:21.123748 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:21.123795 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:21.125059 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:21.125554 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:21.127010 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:21.127187 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:21.127236 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:21.129729 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:21.129730 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:21.129730 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:21.129739 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:21.131373 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:21.131843 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:21.131844 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:21.131845 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:21.131897 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:21.132177 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:21.132177 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:21.132177 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:21.132279 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:21.132326 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:21.132709 (+ 383us) server_negotiation.cc:299] Negotiation successful
0928 06:28:21.132775 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 06:28:21.133139 23944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:21.122730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:39897 (local address 127.22.225.131:47567)
0928 06:28:21.122954 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:21.123005 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:21.123640 (+ 635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:21.123865 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:21.123869 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:21.124666 (+ 797us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:21.124939 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:21.124945 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:21.125638 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:21.125639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:21.126925 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:21.126927 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:21.127262 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:21.127263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:21.127404 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:21.127955 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:21.127965 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:21.129620 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:21.131444 (+ 1824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:21.131450 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:21.131453 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:21.131766 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:21.131958 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:21.131959 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:21.131960 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:21.132108 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:21.132335 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:21.132340 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:21.132524 (+ 184us) client_negotiation.cc:769] Sending connection context
0928 06:28:21.132690 (+ 166us) client_negotiation.cc:240] Negotiation successful
0928 06:28:21.132875 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":101,"threads_started":1}
I20260928 06:28:21.133597 23942 heartbeater.cc:347] Connected to a master server at 127.22.225.190:39897
I20260928 06:28:21.133710 23942 heartbeater.cc:464] Registering TS with master...
I20260928 06:28:21.133890 23942 heartbeater.cc:511] Master 127.22.225.190:39897 requested a full tablet report, sending...
I20260928 06:28:21.134248 23482 ts_manager.cc:194] Registered new tserver with Master: 1ba0cc91ae81409dab5527500151bc8a (127.22.225.131:42447)
I20260928 06:28:21.134636 23482 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.225.131@KRBTEST.COM'} at 127.22.225.131:47567
I20260928 06:28:21.141695 23430 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 06:28:21 dist-test-slave-d59m krb5kdc[23437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576901, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 06:28:21 dist-test-slave-d59m krb5kdc[23437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576901, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.225.129@KRBTEST.COM
I20260928 06:28:21.166492 23949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:21.157613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45220 (local address 127.22.225.129:43311)
0928 06:28:21.157791 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:21.157796 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:21.157807 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:28:21.157834 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:21.157836 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:21.157849 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:21.157916 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:21.158649 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:21.159389 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:21.160436 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:21.161069 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:21.161156 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:21.163405 (+ 2249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:21.163413 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:21.163413 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:21.163419 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:21.165287 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:21.165665 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:21.165682 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:21.165683 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:21.165771 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:21.165990 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:21.165991 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:21.165991 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:21.166135 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:21.166255 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:21.166352 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 06:28:21.166414 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":97,"threads_started":1}
W20260928 06:28:21.166965 23587 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45220
Sep 28 06:28:21 dist-test-slave-d59m krb5kdc[23437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576901, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 06:28:21 dist-test-slave-d59m krb5kdc[23437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576901, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.225.129@KRBTEST.COM
I20260928 06:28:21.191617 23949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:21.183791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45224 (local address 127.22.225.129:43311)
0928 06:28:21.183849 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:21.183851 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:21.183926 (+ 75us) server_negotiation.cc:407] Connection header received
0928 06:28:21.184003 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:21.184005 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:21.184014 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:21.184065 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:21.184798 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:21.185288 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:21.186351 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:21.186550 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:21.186614 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:21.189095 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:21.189097 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:21.189097 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:21.189106 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:21.190483 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:21.190848 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:21.190850 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:21.190850 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:21.190906 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:21.191104 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:21.191105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:21.191105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:21.191205 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:21.191319 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:21.191413 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 06:28:21.191498 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 06:28:21.192152 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 23544
I20260928 06:28:21.197686 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 23678
I20260928 06:28:21.205585 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 23812
I20260928 06:28:21.211393 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 23450
2026-09-28T06:28:21Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2074 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:28:21 dist-test-slave-d59m krb5kdc[23956](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:28:21 dist-test-slave-d59m krb5kdc[23956](info): set up 2 sockets
Sep 28 06:28:21 dist-test-slave-d59m krb5kdc[23956](info): commencing operation
krb5kdc: starting...
W20260928 06:28:22.316548 23430 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Sep 28 06:28:22 dist-test-slave-d59m krb5kdc[23956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:28:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:28:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab.
Entry for principal kudu/127.22.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab.
I20260928 06:28:22.437752 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:44809
--webserver_interface=127.22.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab
--principal=kudu/127.22.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:39845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.225.190:44809
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:28:22.519459 23969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:28:22.519708 23969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:28:22.519743 23969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:28:22.520977 23969 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:28:22.521036 23969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:28:22.521061 23969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:28:22.521082 23969 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:28:22.521114 23969 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:28:22.522655 23969 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:39845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.225.190:44809
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab
--principal=kudu/127.22.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:44809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:28:22.522948 23969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:28:22.523176 23969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:28:22.528106 23977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:22.528126 23975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:22.528106 23974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:22.528389 23969 server_base.cc:1034] running on GCE node
I20260928 06:28:22.528666 23969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:28:22.528975 23969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:28:22.530145 23969 hybrid_clock.cc:648] HybridClock initialized: now 1790576902530122 us; error 41 us; skew 500 ppm
Sep 28 06:28:22 dist-test-slave-d59m krb5kdc[23956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576902, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:28:22.532714 23969 init.cc:377] Logged in from keytab as kudu/127.22.225.190@KRBTEST.COM (short username kudu)
I20260928 06:28:22.533255 23969 webserver.cc:460] Webserver started at http://127.22.225.190:40007/ using document root <none> and password file <none>
I20260928 06:28:22.533509 23969 fs_manager.cc:359] Metadata directory not provided
I20260928 06:28:22.533581 23969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:28:22.533737 23969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:28:22.534876 23969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/minicluster-data/master-0/data/instance:
uuid: "4ccc894621864416972ef99d84473a8f"
format_stamp: "Formatted at 2026-09-28 06:28:22 on dist-test-slave-d59m"
server_key: "5755d1fa9a963a73cae52d74359e1d6d"
server_key_iv: "78f182742f002d9b876f8aa3ce283d20"
server_key_version: "encryptionkey@0"
I20260928 06:28:22.535254 23969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/minicluster-data/master-0/wal/instance:
uuid: "4ccc894621864416972ef99d84473a8f"
format_stamp: "Formatted at 2026-09-28 06:28:22 on dist-test-slave-d59m"
server_key: "5755d1fa9a963a73cae52d74359e1d6d"
server_key_iv: "78f182742f002d9b876f8aa3ce283d20"
server_key_version: "encryptionkey@0"
I20260928 06:28:22.536677 23969 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 06:28:22.537498 23984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:22.537691 23969 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:28:22.537803 23969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/minicluster-data/master-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/minicluster-data/master-0/wal
uuid: "4ccc894621864416972ef99d84473a8f"
format_stamp: "Formatted at 2026-09-28 06:28:22 on dist-test-slave-d59m"
server_key: "5755d1fa9a963a73cae52d74359e1d6d"
server_key_iv: "78f182742f002d9b876f8aa3ce283d20"
server_key_version: "encryptionkey@0"
I20260928 06:28:22.537878 23969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:28:22.557212 23969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:28:22.559676 23969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:28:22.559830 23969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:28:22.564260 23969 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.190:44809
I20260928 06:28:22.564337 24036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.190:44809 every 8 connection(s)
I20260928 06:28:22.564767 23969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
I20260928 06:28:22.565807 24037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:22.569010 24037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ccc894621864416972ef99d84473a8f: Bootstrap starting.
I20260928 06:28:22.569734 24037 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ccc894621864416972ef99d84473a8f: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:22.570062 24037 log.cc:824] T 00000000000000000000000000000000 P 4ccc894621864416972ef99d84473a8f: Log is configured to *not* fsync() on all Append() calls
I20260928 06:28:22.570901 24037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ccc894621864416972ef99d84473a8f: No bootstrap required, opened a new log
I20260928 06:28:22.572767 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 23969
I20260928 06:28:22.572862 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/minicluster-data/master-0/wal/instance
I20260928 06:28:22.573011 23430 external_mini_cluster.cc:1428] Setting key 7d7ffbd0b0bc1059e0cf075e1fb43747
I20260928 06:28:22.573024 24037 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ccc894621864416972ef99d84473a8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ccc894621864416972ef99d84473a8f" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 44809 } }
I20260928 06:28:22.573211 24037 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ccc894621864416972ef99d84473a8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:22.573244 24037 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ccc894621864416972ef99d84473a8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ccc894621864416972ef99d84473a8f, State: Initialized, Role: FOLLOWER
I20260928 06:28:22.573381 24037 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ccc894621864416972ef99d84473a8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ccc894621864416972ef99d84473a8f" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 44809 } }
I20260928 06:28:22.573444 24037 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ccc894621864416972ef99d84473a8f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:28:22.573473 24037 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ccc894621864416972ef99d84473a8f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:28:22.573511 24037 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ccc894621864416972ef99d84473a8f [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:22.574168 24037 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ccc894621864416972ef99d84473a8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ccc894621864416972ef99d84473a8f" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 44809 } }
I20260928 06:28:22.574281 24037 leader_election.cc:302] T 00000000000000000000000000000000 P 4ccc894621864416972ef99d84473a8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ccc894621864416972ef99d84473a8f; no voters:
I20260928 06:28:22.574457 24037 leader_election.cc:288] T 00000000000000000000000000000000 P 4ccc894621864416972ef99d84473a8f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:28:22.574491 24042 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ccc894621864416972ef99d84473a8f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:28:22.574638 24042 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ccc894621864416972ef99d84473a8f [term 1 LEADER]: Becoming Leader. State: Replica: 4ccc894621864416972ef99d84473a8f, State: Running, Role: LEADER
I20260928 06:28:22.574707 24037 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ccc894621864416972ef99d84473a8f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:28:22.574734 24042 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ccc894621864416972ef99d84473a8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ccc894621864416972ef99d84473a8f" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 44809 } }
I20260928 06:28:22.575084 24042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ccc894621864416972ef99d84473a8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ccc894621864416972ef99d84473a8f. Latest consensus state: current_term: 1 leader_uuid: "4ccc894621864416972ef99d84473a8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ccc894621864416972ef99d84473a8f" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 44809 } } }
I20260928 06:28:22.575165 24042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ccc894621864416972ef99d84473a8f [sys.catalog]: This master's current role is: LEADER
I20260928 06:28:22.575325 24043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ccc894621864416972ef99d84473a8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ccc894621864416972ef99d84473a8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ccc894621864416972ef99d84473a8f" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 44809 } } }
I20260928 06:28:22.575394 24043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ccc894621864416972ef99d84473a8f [sys.catalog]: This master's current role is: LEADER
I20260928 06:28:22.575791 24049 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:28:22.576239 24049 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:28:22.577776 24049 catalog_manager.cc:1347] Generated new cluster ID: 90f44c3a628c4607be38b2ea9012b9c2
I20260928 06:28:22.577826 24049 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:28:22 dist-test-slave-d59m krb5kdc[23956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:28:22.585392 24041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:22.573521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45576 (local address 127.22.225.190:44809)
0928 06:28:22.573708 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:22.573718 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:22.573752 (+ 34us) server_negotiation.cc:407] Connection header received
0928 06:28:22.574088 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:22.574094 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:22.574137 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:22.574290 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:22.575624 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:22.577158 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:22.578280 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:22.578906 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:22.578974 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:22.581708 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:22.581712 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:22.581712 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:22.581724 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:22.583709 (+ 1985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:22.584064 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:22.584068 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:22.584068 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:22.584179 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:22.584375 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:22.584375 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:22.584376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:22.584676 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:22.584798 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:22.585027 (+ 229us) server_negotiation.cc:299] Negotiation successful
0928 06:28:22.585185 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":82,"threads_started":1}
I20260928 06:28:22.588121 24049 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:28:22.588704 24049 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:28:22.593555 24049 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ccc894621864416972ef99d84473a8f: Generated new TSK 0
I20260928 06:28:22.593739 24049 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.225.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab.
Entry for principal kudu/127.22.225.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab.
I20260928 06:28:22.623064 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.129:0
--local_ip_for_outbound_sockets=127.22.225.129
--webserver_interface=127.22.225.129
--webserver_port=0
--tserver_master_addrs=127.22.225.190:44809
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab
--principal=kudu/127.22.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:39845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:28:22.706378 24063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:28:22.706637 24063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:28:22.706683 24063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:28:22.708039 24063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:28:22.708107 24063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:28:22.708153 24063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.129
I20260928 06:28:22.709780 24063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:39845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab
--principal=kudu/127.22.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.225.129
--webserver_port=0
--tserver_master_addrs=127.22.225.190:44809
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.129
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:28:22.710078 24063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:28:22.710306 24063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:28:22.715266 24068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:22.715359 24071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:22.715404 24069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:22.715682 24063 server_base.cc:1034] running on GCE node
I20260928 06:28:22.715924 24063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:28:22.716192 24063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:28:22.717381 24063 hybrid_clock.cc:648] HybridClock initialized: now 1790576902717365 us; error 25 us; skew 500 ppm
Sep 28 06:28:22 dist-test-slave-d59m krb5kdc[23956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576902, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:28:22.719923 24063 init.cc:377] Logged in from keytab as kudu/127.22.225.129@KRBTEST.COM (short username kudu)
I20260928 06:28:22.720484 24063 webserver.cc:460] Webserver started at http://127.22.225.129:39847/ using document root <none> and password file <none>
I20260928 06:28:22.720772 24063 fs_manager.cc:359] Metadata directory not provided
I20260928 06:28:22.720841 24063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:28:22.720993 24063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:28:22.722062 24063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/minicluster-data/ts-0/data/instance:
uuid: "9892eb54041844c2aa70771c832d6fe7"
format_stamp: "Formatted at 2026-09-28 06:28:22 on dist-test-slave-d59m"
server_key: "16e24fe96bdb1953c298ec0727b50c79"
server_key_iv: "2346fcada9bc3b5f6fad3cc30511e8f9"
server_key_version: "encryptionkey@0"
I20260928 06:28:22.722429 24063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/minicluster-data/ts-0/wal/instance:
uuid: "9892eb54041844c2aa70771c832d6fe7"
format_stamp: "Formatted at 2026-09-28 06:28:22 on dist-test-slave-d59m"
server_key: "16e24fe96bdb1953c298ec0727b50c79"
server_key_iv: "2346fcada9bc3b5f6fad3cc30511e8f9"
server_key_version: "encryptionkey@0"
I20260928 06:28:22.723774 24063 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:28:22.724696 24078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:22.724850 24063 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:28:22.724936 24063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/minicluster-data/ts-0/wal
uuid: "9892eb54041844c2aa70771c832d6fe7"
format_stamp: "Formatted at 2026-09-28 06:28:22 on dist-test-slave-d59m"
server_key: "16e24fe96bdb1953c298ec0727b50c79"
server_key_iv: "2346fcada9bc3b5f6fad3cc30511e8f9"
server_key_version: "encryptionkey@0"
I20260928 06:28:22.725023 24063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:28:22.739408 24063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:28:22.741977 24063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:28:22.742138 24063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:28:22.742406 24063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:28:22.742770 24063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:28:22.742820 24063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:22.742861 24063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:28:22.742877 24063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:22.749778 24063 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.129:37951
I20260928 06:28:22.749845 24191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.129:37951 every 8 connection(s)
I20260928 06:28:22.750197 24063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
Sep 28 06:28:22 dist-test-slave-d59m krb5kdc[23956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576902, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.129@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:28:22.758015 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 24063
I20260928 06:28:22.758128 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/minicluster-data/ts-0/wal/instance
I20260928 06:28:22.758296 23430 external_mini_cluster.cc:1428] Setting key 3cc865c341f13379e8b2c62d0d9f2653
I20260928 06:28:22.760030 24041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:22.750930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:57325 (local address 127.22.225.190:44809)
0928 06:28:22.750973 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:22.750975 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:22.751235 (+ 260us) server_negotiation.cc:407] Connection header received
0928 06:28:22.752188 (+ 953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:22.752190 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:22.752204 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:22.752261 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:22.752933 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:22.753456 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:22.754490 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:22.754698 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:22.754771 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:22.757056 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:22.757058 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:22.757058 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:22.757067 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:22.758620 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:22.759062 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:22.759064 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:22.759064 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:22.759121 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:22.759387 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:22.759388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:22.759388 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:22.759513 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:22.759583 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:22.759860 (+ 277us) server_negotiation.cc:299] Negotiation successful
0928 06:28:22.759944 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:28:22.760227 24194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:22.750926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:44809 (local address 127.22.225.129:57325)
0928 06:28:22.751098 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:22.751134 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:22.752078 (+ 944us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:22.752329 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:22.752334 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:22.752641 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:22.752840 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:22.752845 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:22.753561 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:22.753562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:22.754394 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:22.754396 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:22.754789 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:22.754790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:22.754911 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:22.755439 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:22.755447 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:22.756947 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:22.758694 (+ 1747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:22.758700 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:22.758702 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:22.758990 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:22.759203 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:22.759204 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:22.759204 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:22.759312 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:22.759589 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:22.759595 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:22.759726 (+ 131us) client_negotiation.cc:769] Sending connection context
0928 06:28:22.759826 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 06:28:22.759987 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":86,"threads_started":1}
WARNING: no policy specified for kudu/127.22.225.130@KRBTEST.COM; defaulting to no policy
I20260928 06:28:22.767163 24192 heartbeater.cc:347] Connected to a master server at 127.22.225.190:44809
I20260928 06:28:22.767288 24192 heartbeater.cc:464] Registering TS with master...
I20260928 06:28:22.767499 24192 heartbeater.cc:511] Master 127.22.225.190:44809 requested a full tablet report, sending...
I20260928 06:28:22.768168 24001 ts_manager.cc:194] Registered new tserver with Master: 9892eb54041844c2aa70771c832d6fe7 (127.22.225.129:37951)
I20260928 06:28:22.768915 24001 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.225.129@KRBTEST.COM'} at 127.22.225.129:57325
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab.
Entry for principal kudu/127.22.225.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab.
I20260928 06:28:22.787206 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.130:0
--local_ip_for_outbound_sockets=127.22.225.130
--webserver_interface=127.22.225.130
--webserver_port=0
--tserver_master_addrs=127.22.225.190:44809
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab
--principal=kudu/127.22.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:39845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:28:22.890022 24197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:28:22.890214 24197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:28:22.890239 24197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:28:22.891360 24197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:28:22.891417 24197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:28:22.891444 24197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.130
I20260928 06:28:22.893090 24197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:39845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab
--principal=kudu/127.22.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.225.130
--webserver_port=0
--tserver_master_addrs=127.22.225.190:44809
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.130
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:28:22.893332 24197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:28:22.893565 24197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:28:22.898422 24203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:22.898452 24202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:22.898613 24205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:22.898669 24197 server_base.cc:1034] running on GCE node
I20260928 06:28:22.898891 24197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:28:22.899132 24197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:28:22.900292 24197 hybrid_clock.cc:648] HybridClock initialized: now 1790576902900281 us; error 34 us; skew 500 ppm
Sep 28 06:28:22 dist-test-slave-d59m krb5kdc[23956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576902, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:28:22.902925 24197 init.cc:377] Logged in from keytab as kudu/127.22.225.130@KRBTEST.COM (short username kudu)
I20260928 06:28:22.903437 24197 webserver.cc:460] Webserver started at http://127.22.225.130:33479/ using document root <none> and password file <none>
I20260928 06:28:22.903687 24197 fs_manager.cc:359] Metadata directory not provided
I20260928 06:28:22.903756 24197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:28:22.903906 24197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:28:22.905000 24197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/minicluster-data/ts-1/data/instance:
uuid: "f32749fe8a294aa9ace1f075b22d923d"
format_stamp: "Formatted at 2026-09-28 06:28:22 on dist-test-slave-d59m"
server_key: "1d448a49f4103b47db9694ac09f6c60b"
server_key_iv: "48507682053ed4fd0775e82920d9e472"
server_key_version: "encryptionkey@0"
I20260928 06:28:22.905344 24197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/minicluster-data/ts-1/wal/instance:
uuid: "f32749fe8a294aa9ace1f075b22d923d"
format_stamp: "Formatted at 2026-09-28 06:28:22 on dist-test-slave-d59m"
server_key: "1d448a49f4103b47db9694ac09f6c60b"
server_key_iv: "48507682053ed4fd0775e82920d9e472"
server_key_version: "encryptionkey@0"
I20260928 06:28:22.906734 24197 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:28:22.907506 24212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:22.907689 24197 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:28:22.907792 24197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/minicluster-data/ts-1/wal
uuid: "f32749fe8a294aa9ace1f075b22d923d"
format_stamp: "Formatted at 2026-09-28 06:28:22 on dist-test-slave-d59m"
server_key: "1d448a49f4103b47db9694ac09f6c60b"
server_key_iv: "48507682053ed4fd0775e82920d9e472"
server_key_version: "encryptionkey@0"
I20260928 06:28:22.907863 24197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:28:22.931802 24197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:28:22.934242 24197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:28:22.934433 24197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:28:22.934710 24197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:28:22.935035 24197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:28:22.935091 24197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:22.935135 24197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:28:22.935159 24197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:22.942480 24197 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.130:38697
I20260928 06:28:22.942530 24325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.130:38697 every 8 connection(s)
I20260928 06:28:22.942929 24197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
I20260928 06:28:22.944811 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 24197
I20260928 06:28:22.944887 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/minicluster-data/ts-1/wal/instance
I20260928 06:28:22.945037 23430 external_mini_cluster.cc:1428] Setting key 376ea063de3a116df1bcbe8623dcec21
WARNING: no policy specified for kudu/127.22.225.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.131@KRBTEST.COM" created.
Sep 28 06:28:23 dist-test-slave-d59m krb5kdc[23956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576902, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.130@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:28:23.080349 24041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:22.943804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:49171 (local address 127.22.225.190:44809)
0928 06:28:22.943881 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:22.943883 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:22.944139 (+ 256us) server_negotiation.cc:407] Connection header received
0928 06:28:22.945600 (+ 1461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:22.945601 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:22.945615 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:22.945684 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:22.946566 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:22.947368 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:22.948927 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:22.949236 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:22.949280 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:23.076428 (+127148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:23.076431 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:23.076431 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:23.076441 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:23.078335 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:23.079053 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:23.079054 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:23.079055 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:23.079126 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:23.079486 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:23.079487 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:23.079488 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:23.079636 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:23.079702 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:23.080136 (+ 434us) server_negotiation.cc:299] Negotiation successful
0928 06:28:23.080248 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 06:28:23.080633 24328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:22.943779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:44809 (local address 127.22.225.130:49171)
0928 06:28:22.943982 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:22.944032 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:22.945464 (+ 1432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:22.945754 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:22.945758 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:22.946157 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:22.946443 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:22.946450 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:22.947475 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:22.947476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:22.948824 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:22.948827 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:22.949373 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:22.949373 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:22.949530 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:22.961095 (+ 11565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:22.961108 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:23.076279 (+115171us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:23.078515 (+ 2236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:23.078545 (+ 30us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:23.078549 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:23.078952 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:23.079203 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:23.079205 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:23.079205 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:23.079387 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:23.079711 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:23.079718 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:23.079937 (+ 219us) client_negotiation.cc:769] Sending connection context
0928 06:28:23.080087 (+ 150us) client_negotiation.cc:240] Negotiation successful
0928 06:28:23.080300 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":111,"threads_started":1}
I20260928 06:28:23.081243 24326 heartbeater.cc:347] Connected to a master server at 127.22.225.190:44809
I20260928 06:28:23.081372 24326 heartbeater.cc:464] Registering TS with master...
I20260928 06:28:23.081602 24326 heartbeater.cc:511] Master 127.22.225.190:44809 requested a full tablet report, sending...
I20260928 06:28:23.082070 24001 ts_manager.cc:194] Registered new tserver with Master: f32749fe8a294aa9ace1f075b22d923d (127.22.225.130:38697)
I20260928 06:28:23.082538 24001 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.225.130@KRBTEST.COM'} at 127.22.225.130:49171
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab.
Entry for principal kudu/127.22.225.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab.
W20260928 06:28:23.304618 23430 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.22.225.131: real 0.355s user 0.003s sys 0.000s
I20260928 06:28:23.304980 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.131:0
--local_ip_for_outbound_sockets=127.22.225.131
--webserver_interface=127.22.225.131
--webserver_port=0
--tserver_master_addrs=127.22.225.190:44809
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab
--principal=kudu/127.22.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:39845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:28:23.388100 24331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:28:23.388336 24331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:28:23.388374 24331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:28:23.389569 24331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:28:23.389638 24331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:28:23.389685 24331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.131
I20260928 06:28:23.391194 24331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:39845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab
--principal=kudu/127.22.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.225.131
--webserver_port=0
--tserver_master_addrs=127.22.225.190:44809
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.131
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:28:23.391466 24331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:28:23.391696 24331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:28:23.396661 24339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:23.396775 24331 server_base.cc:1034] running on GCE node
W20260928 06:28:23.396656 24337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:23.396867 24336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:23.397092 24331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:28:23.397357 24331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:28:23.398510 24331 hybrid_clock.cc:648] HybridClock initialized: now 1790576903398484 us; error 44 us; skew 500 ppm
Sep 28 06:28:23 dist-test-slave-d59m krb5kdc[23956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576903, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:28:23.401072 24331 init.cc:377] Logged in from keytab as kudu/127.22.225.131@KRBTEST.COM (short username kudu)
I20260928 06:28:23.401629 24331 webserver.cc:460] Webserver started at http://127.22.225.131:43917/ using document root <none> and password file <none>
I20260928 06:28:23.401865 24331 fs_manager.cc:359] Metadata directory not provided
I20260928 06:28:23.401911 24331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:28:23.402063 24331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:28:23.403115 24331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/minicluster-data/ts-2/data/instance:
uuid: "29d2eb269378408190f6ce8490ea2ea3"
format_stamp: "Formatted at 2026-09-28 06:28:23 on dist-test-slave-d59m"
server_key: "391b6551f943563f21e59fcb06bc161b"
server_key_iv: "02240ceae8a8d680e9995166919ff939"
server_key_version: "encryptionkey@0"
I20260928 06:28:23.403478 24331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/minicluster-data/ts-2/wal/instance:
uuid: "29d2eb269378408190f6ce8490ea2ea3"
format_stamp: "Formatted at 2026-09-28 06:28:23 on dist-test-slave-d59m"
server_key: "391b6551f943563f21e59fcb06bc161b"
server_key_iv: "02240ceae8a8d680e9995166919ff939"
server_key_version: "encryptionkey@0"
I20260928 06:28:23.404942 24331 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 06:28:23.405741 24346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:23.405931 24331 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:28:23.406035 24331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/minicluster-data/ts-2/wal
uuid: "29d2eb269378408190f6ce8490ea2ea3"
format_stamp: "Formatted at 2026-09-28 06:28:23 on dist-test-slave-d59m"
server_key: "391b6551f943563f21e59fcb06bc161b"
server_key_iv: "02240ceae8a8d680e9995166919ff939"
server_key_version: "encryptionkey@0"
I20260928 06:28:23.406121 24331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:28:23.416213 24331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:28:23.418588 24331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:28:23.418745 24331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:28:23.418968 24331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:28:23.419308 24331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:28:23.419389 24331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:23.419428 24331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:28:23.419456 24331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:23.426751 24331 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.131:42397
I20260928 06:28:23.426800 24459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.131:42397 every 8 connection(s)
I20260928 06:28:23.427129 24331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
I20260928 06:28:23.429898 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 24331
I20260928 06:28:23.429996 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576899143097-23430-0/minicluster-data/ts-2/wal/instance
I20260928 06:28:23.430182 23430 external_mini_cluster.cc:1428] Setting key 13314f7bd3697c150bcfb5e12c963c31
Sep 28 06:28:23 dist-test-slave-d59m krb5kdc[23956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576903, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.131@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:28:23.437431 24041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:23.427864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:46071 (local address 127.22.225.190:44809)
0928 06:28:23.427910 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:23.427912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:23.428149 (+ 237us) server_negotiation.cc:407] Connection header received
0928 06:28:23.429306 (+ 1157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:23.429307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:23.429321 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:23.429375 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:23.430171 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:23.430721 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:23.431719 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:23.431933 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:23.432021 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:23.434530 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:23.434532 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:23.434532 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:23.434539 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:23.436027 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:23.436456 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:23.436457 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:23.436457 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:23.436516 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:23.436793 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:23.436794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:23.436795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:23.436909 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:23.436987 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:23.437265 (+ 278us) server_negotiation.cc:299] Negotiation successful
0928 06:28:23.437343 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:28:23.437618 24462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:23.427851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:44809 (local address 127.22.225.131:46071)
0928 06:28:23.428008 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:23.428043 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:23.429186 (+ 1143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:23.429448 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:23.429457 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:23.429712 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:23.430046 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:23.430052 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:23.430819 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:23.430819 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:23.431624 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:23.431626 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:23.432031 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:23.432032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:23.432156 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:23.432901 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:23.432910 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:23.434422 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:23.436132 (+ 1710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:23.436138 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:23.436140 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:23.436367 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:23.436606 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:23.436607 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:23.436607 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:23.436723 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:23.436977 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:23.436983 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:23.437128 (+ 145us) client_negotiation.cc:769] Sending connection context
0928 06:28:23.437228 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 06:28:23.437383 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":80,"threads_started":1}
I20260928 06:28:23.438171 24460 heartbeater.cc:347] Connected to a master server at 127.22.225.190:44809
I20260928 06:28:23.438275 24460 heartbeater.cc:464] Registering TS with master...
I20260928 06:28:23.438468 24460 heartbeater.cc:511] Master 127.22.225.190:44809 requested a full tablet report, sending...
I20260928 06:28:23.438841 24001 ts_manager.cc:194] Registered new tserver with Master: 29d2eb269378408190f6ce8490ea2ea3 (127.22.225.131:42397)
I20260928 06:28:23.439303 24001 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.225.131@KRBTEST.COM'} at 127.22.225.131:46071
I20260928 06:28:23.445495 23430 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:28:23.453727 24041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:23.447282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45578 (local address 127.22.225.190:44809)
0928 06:28:23.447348 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:23.447351 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:23.447461 (+ 110us) server_negotiation.cc:407] Connection header received
0928 06:28:23.447538 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:23.447539 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:23.447558 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:23.447628 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:23.448202 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:23.448737 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:23.449484 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:23.449663 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:23.449731 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:23.450954 (+ 1223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:23.450955 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:23.450955 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:23.450963 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:23.452448 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:23.452852 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:23.452853 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:23.452853 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:23.452912 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:23.453115 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:23.453116 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:23.453117 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:23.453261 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:23.453383 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:23.453546 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 06:28:23.453624 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 06:28:23.456056 24001 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45578:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:28:23.458276 24001 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:28:23.466029 24475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:23.460999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.129:37951 (local address 127.0.0.1:56090)
0928 06:28:23.461379 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:23.461387 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:23.461429 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:23.461706 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:23.461709 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:23.461711 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:28:23.461778 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:23.461781 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:23.463195 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:23.463196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:23.464677 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:23.464680 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:23.465725 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:23.465726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:23.465815 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:23.465827 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 06:28:23.465904 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 06:28:23.465952 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":119,"thread_start_us":83,"threads_started":1}
I20260928 06:28:23.466475 24474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:23.461098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56090 (local address 127.22.225.129:37951)
0928 06:28:23.461265 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:23.461270 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:23.461447 (+ 177us) server_negotiation.cc:407] Connection header received
0928 06:28:23.461526 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:23.461529 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:23.461550 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:23.461623 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:28:23.461860 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:23.463101 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:23.464770 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:23.465570 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:23.465641 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:23.465692 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:23.466331 (+ 639us) server_negotiation.cc:299] Negotiation successful
0928 06:28:23.466386 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":65,"threads_started":1}
I20260928 06:28:23.466885 24472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:23.460861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46136 (local address 127.22.225.131:42397)
0928 06:28:23.461695 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:23.461700 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:23.461709 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:28:23.461744 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:23.461745 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:23.461759 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:23.461836 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:28:23.461942 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:23.463060 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:23.464948 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:23.466205 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:23.466646 (+ 441us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:23.466705 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:23.466730 (+ 25us) server_negotiation.cc:299] Negotiation successful
0928 06:28:23.466781 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":801,"thread_start_us":86,"threads_started":1}
I20260928 06:28:23.467295 24471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:23.461192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.130:38697 (local address 127.0.0.1:56708)
0928 06:28:23.461519 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:23.461529 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:23.461578 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:23.463666 (+ 2088us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:23.463667 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:23.463669 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:28:23.463703 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:23.463704 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:23.465032 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:23.465033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:23.466116 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:23.466119 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:23.467090 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:23.467091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:23.467148 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:23.467151 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:28:23.467196 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 06:28:23.467233 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":68,"threads_started":1}
I20260928 06:28:23.467514 24476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:23.461349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56708 (local address 127.22.225.130:38697)
0928 06:28:23.463438 (+ 2089us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:23.463442 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:23.463453 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:28:23.463495 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:23.463497 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:23.463512 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:23.463588 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:28:23.463810 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:23.464904 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:23.466206 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:23.466932 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:23.467008 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:23.467056 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:23.467398 (+ 342us) server_negotiation.cc:299] Negotiation successful
0928 06:28:23.467441 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2057,"thread_start_us":87,"threads_started":1}
I20260928 06:28:23.467388 24126 tablet_service.cc:1467] Processing CreateTablet for tablet 0409e5cf5638452dabf3ddea1c0798c2 (DEFAULT_TABLE table=test-table [id=6d953114365e465cb4e6800bad843e7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:28:23.467746 24126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0409e5cf5638452dabf3ddea1c0798c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:23.468185 24260 tablet_service.cc:1467] Processing CreateTablet for tablet 0409e5cf5638452dabf3ddea1c0798c2 (DEFAULT_TABLE table=test-table [id=6d953114365e465cb4e6800bad843e7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:28:23.468470 24260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0409e5cf5638452dabf3ddea1c0798c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:23.466574 24473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:23.460767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.131:42397 (local address 127.0.0.1:46136)
0928 06:28:23.461157 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:23.461201 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:23.461269 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:23.461835 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:23.461836 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:23.461838 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:28:23.461887 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:23.461888 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:23.463183 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:23.463185 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:23.464838 (+ 1653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:23.464842 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:23.466293 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:23.466294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:23.466377 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:23.466381 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:28:23.466409 (+ 28us) client_negotiation.cc:240] Negotiation successful
0928 06:28:23.466466 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"mutex_wait_us":30,"thread_start_us":47,"threads_started":1}
I20260928 06:28:23.471585 24478 tablet_bootstrap.cc:492] T 0409e5cf5638452dabf3ddea1c0798c2 P f32749fe8a294aa9ace1f075b22d923d: Bootstrap starting.
I20260928 06:28:23.472142 24477 tablet_bootstrap.cc:492] T 0409e5cf5638452dabf3ddea1c0798c2 P 9892eb54041844c2aa70771c832d6fe7: Bootstrap starting.
I20260928 06:28:23.472303 24478 tablet_bootstrap.cc:654] T 0409e5cf5638452dabf3ddea1c0798c2 P f32749fe8a294aa9ace1f075b22d923d: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:23.472632 24394 tablet_service.cc:1467] Processing CreateTablet for tablet 0409e5cf5638452dabf3ddea1c0798c2 (DEFAULT_TABLE table=test-table [id=6d953114365e465cb4e6800bad843e7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:28:23.472883 24477 tablet_bootstrap.cc:654] T 0409e5cf5638452dabf3ddea1c0798c2 P 9892eb54041844c2aa70771c832d6fe7: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:23.472965 24394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0409e5cf5638452dabf3ddea1c0798c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:23.472945 24478 log.cc:824] T 0409e5cf5638452dabf3ddea1c0798c2 P f32749fe8a294aa9ace1f075b22d923d: Log is configured to *not* fsync() on all Append() calls
I20260928 06:28:23.473213 24477 log.cc:824] T 0409e5cf5638452dabf3ddea1c0798c2 P 9892eb54041844c2aa70771c832d6fe7: Log is configured to *not* fsync() on all Append() calls
I20260928 06:28:23.473723 24478 tablet_bootstrap.cc:492] T 0409e5cf5638452dabf3ddea1c0798c2 P f32749fe8a294aa9ace1f075b22d923d: No bootstrap required, opened a new log
I20260928 06:28:23.473785 24478 ts_tablet_manager.cc:1397] T 0409e5cf5638452dabf3ddea1c0798c2 P f32749fe8a294aa9ace1f075b22d923d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:28:23.473903 24477 tablet_bootstrap.cc:492] T 0409e5cf5638452dabf3ddea1c0798c2 P 9892eb54041844c2aa70771c832d6fe7: No bootstrap required, opened a new log
I20260928 06:28:23.473951 24477 ts_tablet_manager.cc:1397] T 0409e5cf5638452dabf3ddea1c0798c2 P 9892eb54041844c2aa70771c832d6fe7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:28:23.475868 24478 raft_consensus.cc:348] T 0409e5cf5638452dabf3ddea1c0798c2 P f32749fe8a294aa9ace1f075b22d923d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d2eb269378408190f6ce8490ea2ea3" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 42397 } } peers { permanent_uuid: "9892eb54041844c2aa70771c832d6fe7" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 37951 } } peers { permanent_uuid: "f32749fe8a294aa9ace1f075b22d923d" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 38697 } }
I20260928 06:28:23.476042 24478 raft_consensus.cc:374] T 0409e5cf5638452dabf3ddea1c0798c2 P f32749fe8a294aa9ace1f075b22d923d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:23.476075 24478 raft_consensus.cc:729] T 0409e5cf5638452dabf3ddea1c0798c2 P f32749fe8a294aa9ace1f075b22d923d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f32749fe8a294aa9ace1f075b22d923d, State: Initialized, Role: FOLLOWER
I20260928 06:28:23.476187 24478 consensus_queue.cc:260] T 0409e5cf5638452dabf3ddea1c0798c2 P f32749fe8a294aa9ace1f075b22d923d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d2eb269378408190f6ce8490ea2ea3" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 42397 } } peers { permanent_uuid: "9892eb54041844c2aa70771c832d6fe7" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 37951 } } peers { permanent_uuid: "f32749fe8a294aa9ace1f075b22d923d" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 38697 } }
I20260928 06:28:23.476456 24478 ts_tablet_manager.cc:1428] T 0409e5cf5638452dabf3ddea1c0798c2 P f32749fe8a294aa9ace1f075b22d923d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:28:23.476640 24326 heartbeater.cc:503] Master 127.22.225.190:44809 was elected leader, sending a full tablet report...
I20260928 06:28:23.476619 24477 raft_consensus.cc:348] T 0409e5cf5638452dabf3ddea1c0798c2 P 9892eb54041844c2aa70771c832d6fe7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d2eb269378408190f6ce8490ea2ea3" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 42397 } } peers { permanent_uuid: "9892eb54041844c2aa70771c832d6fe7" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 37951 } } peers { permanent_uuid: "f32749fe8a294aa9ace1f075b22d923d" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 38697 } }
I20260928 06:28:23.476769 24477 raft_consensus.cc:374] T 0409e5cf5638452dabf3ddea1c0798c2 P 9892eb54041844c2aa70771c832d6fe7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:23.476761 24481 tablet_bootstrap.cc:492] T 0409e5cf5638452dabf3ddea1c0798c2 P 29d2eb269378408190f6ce8490ea2ea3: Bootstrap starting.
I20260928 06:28:23.476806 24477 raft_consensus.cc:729] T 0409e5cf5638452dabf3ddea1c0798c2 P 9892eb54041844c2aa70771c832d6fe7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9892eb54041844c2aa70771c832d6fe7, State: Initialized, Role: FOLLOWER
I20260928 06:28:23.476971 24477 consensus_queue.cc:260] T 0409e5cf5638452dabf3ddea1c0798c2 P 9892eb54041844c2aa70771c832d6fe7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d2eb269378408190f6ce8490ea2ea3" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 42397 } } peers { permanent_uuid: "9892eb54041844c2aa70771c832d6fe7" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 37951 } } peers { permanent_uuid: "f32749fe8a294aa9ace1f075b22d923d" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 38697 } }
I20260928 06:28:23.477223 24477 ts_tablet_manager.cc:1428] T 0409e5cf5638452dabf3ddea1c0798c2 P 9892eb54041844c2aa70771c832d6fe7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:28:23.477324 24192 heartbeater.cc:503] Master 127.22.225.190:44809 was elected leader, sending a full tablet report...
I20260928 06:28:23.477386 24481 tablet_bootstrap.cc:654] T 0409e5cf5638452dabf3ddea1c0798c2 P 29d2eb269378408190f6ce8490ea2ea3: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:23.477931 24481 log.cc:824] T 0409e5cf5638452dabf3ddea1c0798c2 P 29d2eb269378408190f6ce8490ea2ea3: Log is configured to *not* fsync() on all Append() calls
I20260928 06:28:23.478672 24481 tablet_bootstrap.cc:492] T 0409e5cf5638452dabf3ddea1c0798c2 P 29d2eb269378408190f6ce8490ea2ea3: No bootstrap required, opened a new log
I20260928 06:28:23.478741 24481 ts_tablet_manager.cc:1397] T 0409e5cf5638452dabf3ddea1c0798c2 P 29d2eb269378408190f6ce8490ea2ea3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:28:23.480667 24481 raft_consensus.cc:348] T 0409e5cf5638452dabf3ddea1c0798c2 P 29d2eb269378408190f6ce8490ea2ea3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d2eb269378408190f6ce8490ea2ea3" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 42397 } } peers { permanent_uuid: "9892eb54041844c2aa70771c832d6fe7" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 37951 } } peers { permanent_uuid: "f32749fe8a294aa9ace1f075b22d923d" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 38697 } }
I20260928 06:28:23.480836 24481 raft_consensus.cc:374] T 0409e5cf5638452dabf3ddea1c0798c2 P 29d2eb269378408190f6ce8490ea2ea3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:23.480876 24481 raft_consensus.cc:729] T 0409e5cf5638452dabf3ddea1c0798c2 P 29d2eb269378408190f6ce8490ea2ea3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d2eb269378408190f6ce8490ea2ea3, State: Initialized, Role: FOLLOWER
I20260928 06:28:23.481002 24481 consensus_queue.cc:260] T 0409e5cf5638452dabf3ddea1c0798c2 P 29d2eb269378408190f6ce8490ea2ea3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d2eb269378408190f6ce8490ea2ea3" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 42397 } } peers { permanent_uuid: "9892eb54041844c2aa70771c832d6fe7" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 37951 } } peers { permanent_uuid: "f32749fe8a294aa9ace1f075b22d923d" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 38697 } }
I20260928 06:28:23.481325 24481 ts_tablet_manager.cc:1428] T 0409e5cf5638452dabf3ddea1c0798c2 P 29d2eb269378408190f6ce8490ea2ea3: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 06:28:23.481329 24460 heartbeater.cc:503] Master 127.22.225.190:44809 was elected leader, sending a full tablet report...
W20260928 06:28:23.501113 24193 tablet.cc:2369] T 0409e5cf5638452dabf3ddea1c0798c2 P 9892eb54041844c2aa70771c832d6fe7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:28:23.525664 24483 raft_consensus.cc:482] T 0409e5cf5638452dabf3ddea1c0798c2 P 9892eb54041844c2aa70771c832d6fe7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:28:23.525816 24483 raft_consensus.cc:504] T 0409e5cf5638452dabf3ddea1c0798c2 P 9892eb54041844c2aa70771c832d6fe7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d2eb269378408190f6ce8490ea2ea3" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 42397 } } peers { permanent_uuid: "9892eb54041844c2aa70771c832d6fe7" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 37951 } } peers { permanent_uuid: "f32749fe8a294aa9ace1f075b22d923d" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 38697 } }
I20260928 06:28:23.526237 24483 leader_election.cc:288] T 0409e5cf5638452dabf3ddea1c0798c2 P 9892eb54041844c2aa70771c832d6fe7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29d2eb269378408190f6ce8490ea2ea3 (127.22.225.131:42397), f32749fe8a294aa9ace1f075b22d923d (127.22.225.130:38697)
I20260928 06:28:23.530124 24472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:23.526517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:35307 (local address 127.22.225.131:42397)
0928 06:28:23.526603 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:23.526606 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:23.526707 (+ 101us) server_negotiation.cc:407] Connection header received
0928 06:28:23.526841 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:23.526842 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:23.526852 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:23.526903 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:28:23.527133 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:23.527885 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:23.529389 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:23.529772 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:23.529835 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:23.529842 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:23.530037 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 06:28:23.530068 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 06:28:23.530223 24486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:23.526461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.131:42397 (local address 127.22.225.129:35307)
0928 06:28:23.526622 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:23.526633 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:23.526724 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:23.526942 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:23.526944 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:23.526946 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:28:23.527054 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:23.527057 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:23.527975 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:23.527976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:23.529297 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:23.529301 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:23.529857 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:23.529859 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:23.529953 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:23.529959 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 06:28:23.530032 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 06:28:23.530088 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":1,"thread_start_us":97,"threads_started":1}
I20260928 06:28:23.530629 24414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0409e5cf5638452dabf3ddea1c0798c2" candidate_uuid: "9892eb54041844c2aa70771c832d6fe7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d2eb269378408190f6ce8490ea2ea3" is_pre_election: true
I20260928 06:28:23.530792 24414 raft_consensus.cc:2454] T 0409e5cf5638452dabf3ddea1c0798c2 P 29d2eb269378408190f6ce8490ea2ea3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9892eb54041844c2aa70771c832d6fe7 in term 0.
I20260928 06:28:23.531054 24080 leader_election.cc:302] T 0409e5cf5638452dabf3ddea1c0798c2 P 9892eb54041844c2aa70771c832d6fe7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d2eb269378408190f6ce8490ea2ea3, 9892eb54041844c2aa70771c832d6fe7; no voters:
I20260928 06:28:23.531211 24483 raft_consensus.cc:2781] T 0409e5cf5638452dabf3ddea1c0798c2 P 9892eb54041844c2aa70771c832d6fe7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:28:23.531286 24483 raft_consensus.cc:482] T 0409e5cf5638452dabf3ddea1c0798c2 P 9892eb54041844c2aa70771c832d6fe7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:28:23.531315 24483 raft_consensus.cc:3037] T 0409e5cf5638452dabf3ddea1c0798c2 P 9892eb54041844c2aa70771c832d6fe7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:23.532181 24487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:23.526704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.130:38697 (local address 127.22.225.129:55209)
0928 06:28:23.526818 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:23.526830 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:23.526899 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:23.527051 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:23.527052 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:23.527053 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:28:23.527115 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:23.527116 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:23.527877 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:23.527878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:23.531085 (+ 3207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:23.531088 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:23.531875 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:23.531876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:23.531954 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:23.531959 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:28:23.532018 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 06:28:23.532066 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"mutex_wait_us":47,"thread_start_us":68,"threads_started":1}
I20260928 06:28:23.532166 24476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:23.526778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:55209 (local address 127.22.225.130:38697)
0928 06:28:23.526841 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:23.526844 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:23.526900 (+ 56us) server_negotiation.cc:407] Connection header received
0928 06:28:23.526975 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:23.526977 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:23.526986 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:23.527041 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:28:23.527197 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:23.527762 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:23.531183 (+ 3421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:23.531780 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:23.531850 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:23.531858 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:23.532039 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 06:28:23.532080 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 06:28:23.532317 24483 raft_consensus.cc:504] T 0409e5cf5638452dabf3ddea1c0798c2 P 9892eb54041844c2aa70771c832d6fe7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d2eb269378408190f6ce8490ea2ea3" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 42397 } } peers { permanent_uuid: "9892eb54041844c2aa70771c832d6fe7" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 37951 } } peers { permanent_uuid: "f32749fe8a294aa9ace1f075b22d923d" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 38697 } }
I20260928 06:28:23.532450 24483 leader_election.cc:288] T 0409e5cf5638452dabf3ddea1c0798c2 P 9892eb54041844c2aa70771c832d6fe7 [CANDIDATE]: Term 1 election: Requested vote from peers 29d2eb269378408190f6ce8490ea2ea3 (127.22.225.131:42397), f32749fe8a294aa9ace1f075b22d923d (127.22.225.130:38697)
I20260928 06:28:23.532487 24280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0409e5cf5638452dabf3ddea1c0798c2" candidate_uuid: "9892eb54041844c2aa70771c832d6fe7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f32749fe8a294aa9ace1f075b22d923d" is_pre_election: true
I20260928 06:28:23.532622 24414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0409e5cf5638452dabf3ddea1c0798c2" candidate_uuid: "9892eb54041844c2aa70771c832d6fe7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d2eb269378408190f6ce8490ea2ea3"
I20260928 06:28:23.532660 24279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0409e5cf5638452dabf3ddea1c0798c2" candidate_uuid: "9892eb54041844c2aa70771c832d6fe7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f32749fe8a294aa9ace1f075b22d923d"
I20260928 06:28:23.532697 24280 raft_consensus.cc:2454] T 0409e5cf5638452dabf3ddea1c0798c2 P f32749fe8a294aa9ace1f075b22d923d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9892eb54041844c2aa70771c832d6fe7 in term 0.
I20260928 06:28:23.532706 24414 raft_consensus.cc:3037] T 0409e5cf5638452dabf3ddea1c0798c2 P 29d2eb269378408190f6ce8490ea2ea3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:23.533656 24414 raft_consensus.cc:2454] T 0409e5cf5638452dabf3ddea1c0798c2 P 29d2eb269378408190f6ce8490ea2ea3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9892eb54041844c2aa70771c832d6fe7 in term 1.
I20260928 06:28:23.533859 24080 leader_election.cc:302] T 0409e5cf5638452dabf3ddea1c0798c2 P 9892eb54041844c2aa70771c832d6fe7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29d2eb269378408190f6ce8490ea2ea3, 9892eb54041844c2aa70771c832d6fe7; no voters: f32749fe8a294aa9ace1f075b22d923d
I20260928 06:28:23.533968 24483 raft_consensus.cc:2781] T 0409e5cf5638452dabf3ddea1c0798c2 P 9892eb54041844c2aa70771c832d6fe7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:28:23.534137 24483 raft_consensus.cc:686] T 0409e5cf5638452dabf3ddea1c0798c2 P 9892eb54041844c2aa70771c832d6fe7 [term 1 LEADER]: Becoming Leader. State: Replica: 9892eb54041844c2aa70771c832d6fe7, State: Running, Role: LEADER
I20260928 06:28:23.534256 24483 consensus_queue.cc:237] T 0409e5cf5638452dabf3ddea1c0798c2 P 9892eb54041844c2aa70771c832d6fe7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d2eb269378408190f6ce8490ea2ea3" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 42397 } } peers { permanent_uuid: "9892eb54041844c2aa70771c832d6fe7" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 37951 } } peers { permanent_uuid: "f32749fe8a294aa9ace1f075b22d923d" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 38697 } }
I20260928 06:28:23.535099 24001 catalog_manager.cc:5495] T 0409e5cf5638452dabf3ddea1c0798c2 P 9892eb54041844c2aa70771c832d6fe7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9892eb54041844c2aa70771c832d6fe7 (127.22.225.129). New cstate: current_term: 1 leader_uuid: "9892eb54041844c2aa70771c832d6fe7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d2eb269378408190f6ce8490ea2ea3" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 42397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9892eb54041844c2aa70771c832d6fe7" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 37951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f32749fe8a294aa9ace1f075b22d923d" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 38697 } health_report { overall_health: UNKNOWN } } }
Sep 28 06:28:23 dist-test-slave-d59m krb5kdc[23956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.225.129@KRBTEST.COM
I20260928 06:28:23.563657 24474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:23.555353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56100 (local address 127.22.225.129:37951)
0928 06:28:23.555406 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:23.555409 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:23.555479 (+ 70us) server_negotiation.cc:407] Connection header received
0928 06:28:23.555560 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:23.555561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:23.555571 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:23.555638 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:23.556311 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:23.556874 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:23.557582 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:23.557784 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:23.557861 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:23.560456 (+ 2595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:23.560470 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:23.560470 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:23.560478 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:23.562551 (+ 2073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:23.562946 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:23.562949 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:23.562950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:23.563019 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:23.563185 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:23.563186 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:23.563186 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:23.563329 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:23.563431 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:23.563483 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 06:28:23.563544 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
Sep 28 06:28:23 dist-test-slave-d59m krb5kdc[23956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576903, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 06:28:23 dist-test-slave-d59m krb5kdc[23956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576903, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.225.129@KRBTEST.COM
I20260928 06:28:23.586022 24474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:23.579103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56116 (local address 127.22.225.129:37951)
0928 06:28:23.579171 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:23.579173 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:23.579262 (+ 89us) server_negotiation.cc:407] Connection header received
0928 06:28:23.579308 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:23.579309 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:23.579319 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:23.579369 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:23.579887 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:23.580357 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:23.581122 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:23.581310 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:23.581366 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:23.583603 (+ 2237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:23.583604 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:23.583604 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:23.583611 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:23.585075 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:23.585369 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:23.585370 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:23.585370 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:23.585419 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:23.585569 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:23.585569 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:23.585569 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:23.585654 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:23.585740 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:23.585845 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 06:28:23.585917 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
Sep 28 06:28:23 dist-test-slave-d59m krb5kdc[23956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 06:28:23 dist-test-slave-d59m krb5kdc[23956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.225.129@KRBTEST.COM
I20260928 06:28:23.609144 24474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:23.601660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56128 (local address 127.22.225.129:37951)
0928 06:28:23.601741 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:23.601743 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:23.601753 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:28:23.601820 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:23.601821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:23.601832 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:23.601926 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:23.602602 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:23.603184 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:23.604218 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:23.604420 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:23.604504 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:23.606661 (+ 2157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:23.606663 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:23.606663 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:23.606671 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:23.608083 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:23.608394 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:23.608395 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:23.608395 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:23.608447 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:23.608668 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:23.608668 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:23.608669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:23.608810 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:23.608902 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:23.608992 (+ 90us) server_negotiation.cc:299] Negotiation successful
0928 06:28:23.609065 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260928 06:28:23.609632 24100 tablet_service.cc:2908] Rejecting scan request for tablet 0409e5cf5638452dabf3ddea1c0798c2: Uninitialized: clean time has not yet been initialized
I20260928 06:28:23.610630 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 24063
I20260928 06:28:23.616814 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 24197
I20260928 06:28:23.622341 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 24331
I20260928 06:28:23.627331 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 23969
2026-09-28T06:28:23Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2416 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:28:23 dist-test-slave-d59m krb5kdc[24502](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:28:23 dist-test-slave-d59m krb5kdc[24502](info): set up 2 sockets
Sep 28 06:28:23 dist-test-slave-d59m krb5kdc[24502](info): commencing operation
krb5kdc: starting...
W20260928 06:28:24.696791 23430 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Sep 28 06:28:24 dist-test-slave-d59m krb5kdc[24502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:28:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:28:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab.
Entry for principal kudu/127.22.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab.
I20260928 06:28:24.819434 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:46621
--webserver_interface=127.22.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab
--principal=kudu/127.22.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:34263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.225.190:46621
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:28:24.939410 24515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:28:24.939697 24515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:28:24.939751 24515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:28:24.941592 24515 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:28:24.941651 24515 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 06:28:24.941672 24515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:28:24.941694 24515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:28:24.941711 24515 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:28:24.941740 24515 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:28:24.943965 24515 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:34263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.225.190:46621
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab
--principal=kudu/127.22.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:46621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:28:24.944226 24515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:28:24.944496 24515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:28:24.949733 24521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:24.949779 24523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:24.950011 24515 server_base.cc:1034] running on GCE node
W20260928 06:28:24.949751 24520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:24.950361 24515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:28:24.950662 24515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:28:24.951824 24515 hybrid_clock.cc:648] HybridClock initialized: now 1790576904951794 us; error 50 us; skew 500 ppm
Sep 28 06:28:24 dist-test-slave-d59m krb5kdc[24502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576904, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:28:24.954344 24515 init.cc:377] Logged in from keytab as kudu/127.22.225.190@KRBTEST.COM (short username kudu)
I20260928 06:28:24.954872 24515 webserver.cc:460] Webserver started at http://127.22.225.190:33103/ using document root <none> and password file <none>
I20260928 06:28:24.955127 24515 fs_manager.cc:359] Metadata directory not provided
I20260928 06:28:24.955196 24515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:28:24.955343 24515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:28:24.956399 24515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/master-0/data/instance:
uuid: "4918f18b857f4b0696388f71b4355706"
format_stamp: "Formatted at 2026-09-28 06:28:24 on dist-test-slave-d59m"
server_key: "6eaf98500b32e0313448a9fc8ea2d2e6"
server_key_iv: "a2f8522fc287a2a11c52eb108baf1343"
server_key_version: "encryptionkey@0"
I20260928 06:28:24.956801 24515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/master-0/wal/instance:
uuid: "4918f18b857f4b0696388f71b4355706"
format_stamp: "Formatted at 2026-09-28 06:28:24 on dist-test-slave-d59m"
server_key: "6eaf98500b32e0313448a9fc8ea2d2e6"
server_key_iv: "a2f8522fc287a2a11c52eb108baf1343"
server_key_version: "encryptionkey@0"
I20260928 06:28:24.958174 24515 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:28:24.958952 24530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:24.959131 24515 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:28:24.959229 24515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/master-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/master-0/wal
uuid: "4918f18b857f4b0696388f71b4355706"
format_stamp: "Formatted at 2026-09-28 06:28:24 on dist-test-slave-d59m"
server_key: "6eaf98500b32e0313448a9fc8ea2d2e6"
server_key_iv: "a2f8522fc287a2a11c52eb108baf1343"
server_key_version: "encryptionkey@0"
I20260928 06:28:24.959308 24515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:28:24.973862 24515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:28:24.976020 24515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:28:24.976203 24515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:28:24.981484 24515 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.190:46621
I20260928 06:28:24.981542 24592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.190:46621 every 8 connection(s)
I20260928 06:28:24.981889 24515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
I20260928 06:28:24.982661 24593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:24.984989 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 24515
I20260928 06:28:24.985090 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/master-0/wal/instance
I20260928 06:28:24.985244 23430 external_mini_cluster.cc:1428] Setting key 4485b27a2118ca1b1e6283d6a488f8cc
I20260928 06:28:24.985782 24593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4918f18b857f4b0696388f71b4355706: Bootstrap starting.
I20260928 06:28:24.986573 24593 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4918f18b857f4b0696388f71b4355706: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:24.986944 24593 log.cc:824] T 00000000000000000000000000000000 P 4918f18b857f4b0696388f71b4355706: Log is configured to *not* fsync() on all Append() calls
I20260928 06:28:24.987684 24593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4918f18b857f4b0696388f71b4355706: No bootstrap required, opened a new log
I20260928 06:28:24.989974 24593 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4918f18b857f4b0696388f71b4355706 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4918f18b857f4b0696388f71b4355706" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 46621 } }
I20260928 06:28:24.990136 24593 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4918f18b857f4b0696388f71b4355706 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:24.990198 24593 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4918f18b857f4b0696388f71b4355706 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4918f18b857f4b0696388f71b4355706, State: Initialized, Role: FOLLOWER
I20260928 06:28:24.990337 24593 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4918f18b857f4b0696388f71b4355706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4918f18b857f4b0696388f71b4355706" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 46621 } }
I20260928 06:28:24.990437 24593 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4918f18b857f4b0696388f71b4355706 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:28:24.990491 24593 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4918f18b857f4b0696388f71b4355706 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:28:24.990592 24593 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4918f18b857f4b0696388f71b4355706 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:24.991335 24593 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4918f18b857f4b0696388f71b4355706 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4918f18b857f4b0696388f71b4355706" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 46621 } }
I20260928 06:28:24.991451 24593 leader_election.cc:302] T 00000000000000000000000000000000 P 4918f18b857f4b0696388f71b4355706 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4918f18b857f4b0696388f71b4355706; no voters:
I20260928 06:28:24.991680 24593 leader_election.cc:288] T 00000000000000000000000000000000 P 4918f18b857f4b0696388f71b4355706 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:28:24.991731 24598 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4918f18b857f4b0696388f71b4355706 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:28:24.991926 24598 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4918f18b857f4b0696388f71b4355706 [term 1 LEADER]: Becoming Leader. State: Replica: 4918f18b857f4b0696388f71b4355706, State: Running, Role: LEADER
I20260928 06:28:24.992022 24593 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4918f18b857f4b0696388f71b4355706 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:28:24.992049 24598 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4918f18b857f4b0696388f71b4355706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4918f18b857f4b0696388f71b4355706" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 46621 } }
I20260928 06:28:24.992460 24598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4918f18b857f4b0696388f71b4355706 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4918f18b857f4b0696388f71b4355706. Latest consensus state: current_term: 1 leader_uuid: "4918f18b857f4b0696388f71b4355706" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4918f18b857f4b0696388f71b4355706" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 46621 } } }
I20260928 06:28:24.992730 24598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4918f18b857f4b0696388f71b4355706 [sys.catalog]: This master's current role is: LEADER
Sep 28 06:28:24 dist-test-slave-d59m krb5kdc[24502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:28:24.992656 24599 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4918f18b857f4b0696388f71b4355706 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4918f18b857f4b0696388f71b4355706" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4918f18b857f4b0696388f71b4355706" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 46621 } } }
I20260928 06:28:24.993098 24599 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4918f18b857f4b0696388f71b4355706 [sys.catalog]: This master's current role is: LEADER
I20260928 06:28:24.993180 24607 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:28:24.993702 24607 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:28:24.997051 24596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:24.985753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34470 (local address 127.22.225.190:46621)
0928 06:28:24.985930 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:24.985945 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:24.985971 (+ 26us) server_negotiation.cc:407] Connection header received
0928 06:28:24.986396 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:24.986403 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:24.986447 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:24.986611 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:24.987288 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:24.988779 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:24.989845 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:24.990472 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:24.990576 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:24.993770 (+ 3194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:24.993774 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:24.993774 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:24.993787 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:24.995523 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:24.995919 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:24.995922 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:24.995922 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:24.996012 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:24.996218 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:24.996219 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:24.996219 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:24.996445 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:24.996647 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:24.996716 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 06:28:24.996853 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":99,"threads_started":1}
I20260928 06:28:25.001269 24607 catalog_manager.cc:1347] Generated new cluster ID: 7ccec952953242f2ace8b36b34b1f39e
I20260928 06:28:25.001332 24607 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:28:25.016342 24607 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:28:25.017210 24607 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:28:25.023882 24607 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4918f18b857f4b0696388f71b4355706: Generated new TSK 0
I20260928 06:28:25.024101 24607 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.225.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab.
Entry for principal kudu/127.22.225.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab.
I20260928 06:28:25.053437 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.129:0
--local_ip_for_outbound_sockets=127.22.225.129
--webserver_interface=127.22.225.129
--webserver_port=0
--tserver_master_addrs=127.22.225.190:46621
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab
--principal=kudu/127.22.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:34263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:28:25.142016 24619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:28:25.142309 24619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:28:25.142345 24619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:28:25.144264 24619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:28:25.144337 24619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:28:25.144376 24619 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 06:28:25.144407 24619 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.129
I20260928 06:28:25.146597 24619 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:34263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab
--principal=kudu/127.22.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.225.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.225.190:46621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.129
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:28:25.146894 24619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:28:25.147158 24619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:28:25.152281 24625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:25.152411 24624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:25.152281 24627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:25.152987 24619 server_base.cc:1034] running on GCE node
I20260928 06:28:25.153206 24619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:28:25.153491 24619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:28:25.154709 24619 hybrid_clock.cc:648] HybridClock initialized: now 1790576905154690 us; error 37 us; skew 500 ppm
Sep 28 06:28:25 dist-test-slave-d59m krb5kdc[24502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576905, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:28:25.157433 24619 init.cc:377] Logged in from keytab as kudu/127.22.225.129@KRBTEST.COM (short username kudu)
I20260928 06:28:25.158041 24619 webserver.cc:460] Webserver started at http://127.22.225.129:36403/ using document root <none> and password file <none>
I20260928 06:28:25.158291 24619 fs_manager.cc:359] Metadata directory not provided
I20260928 06:28:25.158360 24619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:28:25.158505 24619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:28:25.159574 24619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/ts-0/data/instance:
uuid: "59eeadf53cb043659a02cba07a4300ef"
format_stamp: "Formatted at 2026-09-28 06:28:25 on dist-test-slave-d59m"
server_key: "42bddf87b7295223687d0f4097bb514f"
server_key_iv: "c3b0fb9a08fc89ef284990b81eb577e4"
server_key_version: "encryptionkey@0"
I20260928 06:28:25.159953 24619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/ts-0/wal/instance:
uuid: "59eeadf53cb043659a02cba07a4300ef"
format_stamp: "Formatted at 2026-09-28 06:28:25 on dist-test-slave-d59m"
server_key: "42bddf87b7295223687d0f4097bb514f"
server_key_iv: "c3b0fb9a08fc89ef284990b81eb577e4"
server_key_version: "encryptionkey@0"
I20260928 06:28:25.161374 24619 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:28:25.162263 24634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:25.162446 24619 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:28:25.162546 24619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/ts-0/wal
uuid: "59eeadf53cb043659a02cba07a4300ef"
format_stamp: "Formatted at 2026-09-28 06:28:25 on dist-test-slave-d59m"
server_key: "42bddf87b7295223687d0f4097bb514f"
server_key_iv: "c3b0fb9a08fc89ef284990b81eb577e4"
server_key_version: "encryptionkey@0"
I20260928 06:28:25.162635 24619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:28:25.171511 24619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:28:25.173640 24619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:28:25.173799 24619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:28:25.174295 24619 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:28:25.174361 24619 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:25.174444 24619 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:28:25.174497 24619 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 06:28:25 dist-test-slave-d59m krb5kdc[24502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576905, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.129@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:28:25.182173 24619 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.129:43685
I20260928 06:28:25.182284 24749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.129:43685 every 8 connection(s)
I20260928 06:28:25.182654 24619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
I20260928 06:28:25.185535 24643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:25.174778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:46621 (local address 127.22.225.129:40351)
0928 06:28:25.174953 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:25.175007 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:25.175591 (+ 584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:25.175885 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:25.175889 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:25.176264 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:25.176554 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:25.176561 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:25.177622 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:25.177624 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:25.178839 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:25.178842 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:25.179424 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:25.179425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:25.179565 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:25.180234 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:25.180244 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:25.182078 (+ 1834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:25.183999 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:25.184004 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:25.184006 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:25.184227 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:25.184453 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:25.184454 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:25.184454 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:25.184558 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:25.184901 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:25.184916 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:25.185051 (+ 135us) client_negotiation.cc:769] Sending connection context
0928 06:28:25.185174 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 06:28:25.185371 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":84,"threads_started":1}
I20260928 06:28:25.186094 24596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:25.175113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:40351 (local address 127.22.225.190:46621)
0928 06:28:25.175160 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:25.175162 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:25.175175 (+ 13us) server_negotiation.cc:407] Connection header received
0928 06:28:25.175710 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:25.175712 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:25.175726 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:25.175797 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:25.176681 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:25.177513 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:25.178955 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:25.179282 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:25.179343 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:25.182190 (+ 2847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:25.182191 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:25.182192 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:25.182213 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:25.183908 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:25.184306 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:25.184307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:25.184307 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:25.184376 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:25.184681 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:25.184682 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:25.184682 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:25.184830 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:25.184911 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:25.185831 (+ 920us) server_negotiation.cc:299] Negotiation successful
0928 06:28:25.185971 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 06:28:25.186522 24750 heartbeater.cc:347] Connected to a master server at 127.22.225.190:46621
I20260928 06:28:25.186619 24750 heartbeater.cc:464] Registering TS with master...
I20260928 06:28:25.186802 24750 heartbeater.cc:511] Master 127.22.225.190:46621 requested a full tablet report, sending...
I20260928 06:28:25.187680 24546 ts_manager.cc:194] Registered new tserver with Master: 59eeadf53cb043659a02cba07a4300ef (127.22.225.129:43685)
I20260928 06:28:25.188311 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 24619
I20260928 06:28:25.188409 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/ts-0/wal/instance
I20260928 06:28:25.188536 24546 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.225.129@KRBTEST.COM'} at 127.22.225.129:40351
I20260928 06:28:25.188598 23430 external_mini_cluster.cc:1428] Setting key 6897f5ad9d0378094257256abd917b65
I20260928 06:28:25.195608 24596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:25.187585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:33781 (local address 127.22.225.190:46621)
0928 06:28:25.187623 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:25.187625 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:25.187772 (+ 147us) server_negotiation.cc:407] Connection header received
0928 06:28:25.187821 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:25.187822 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:25.187832 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:25.187888 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:25.188274 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:25.189049 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:25.190453 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:25.190732 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:25.190765 (+ 33us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:25.192203 (+ 1438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:25.192205 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:25.192205 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:25.192212 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:25.194407 (+ 2195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:25.194782 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:25.194784 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:25.194784 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:25.194840 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:25.195090 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:25.195090 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:25.195091 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:25.195204 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:25.195289 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:25.195443 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 06:28:25.195515 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:28:25.195613 24756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:25.187544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:46621 (local address 127.22.225.129:33781)
0928 06:28:25.187690 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:25.187702 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:25.187773 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:25.187895 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:25.187896 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:25.188084 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:25.188173 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:25.188175 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:25.189251 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:25.189252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:25.190336 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:25.190339 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:25.190856 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:25.190857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:25.190937 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:25.191706 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:25.191738 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:25.192105 (+ 367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:25.194522 (+ 2417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:25.194523 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:25.194524 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:25.194699 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:25.194956 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:25.194957 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:25.194957 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:25.195020 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:25.195280 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:25.195281 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:25.195327 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 06:28:25.195409 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 06:28:25.195516 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":89,"threads_started":1}
WARNING: no policy specified for kudu/127.22.225.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab.
Entry for principal kudu/127.22.225.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab.
I20260928 06:28:25.215468 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.130:0
--local_ip_for_outbound_sockets=127.22.225.130
--webserver_interface=127.22.225.130
--webserver_port=0
--tserver_master_addrs=127.22.225.190:46621
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab
--principal=kudu/127.22.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:34263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:28:25.303005 24759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:28:25.303293 24759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:28:25.303335 24759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:28:25.304955 24759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:28:25.305016 24759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:28:25.305052 24759 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 06:28:25.305089 24759 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.130
I20260928 06:28:25.306638 24759 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:34263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab
--principal=kudu/127.22.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.225.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.225.190:46621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.130
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:28:25.306895 24759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:28:25.307158 24759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:28:25.313249 24767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:25.313231 24764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:25.313231 24765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:25.313704 24759 server_base.cc:1034] running on GCE node
I20260928 06:28:25.313962 24759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:28:25.314244 24759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:28:25.315423 24759 hybrid_clock.cc:648] HybridClock initialized: now 1790576905315391 us; error 45 us; skew 500 ppm
Sep 28 06:28:25 dist-test-slave-d59m krb5kdc[24502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576905, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:28:25.318184 24759 init.cc:377] Logged in from keytab as kudu/127.22.225.130@KRBTEST.COM (short username kudu)
I20260928 06:28:25.318702 24759 webserver.cc:460] Webserver started at http://127.22.225.130:39267/ using document root <none> and password file <none>
I20260928 06:28:25.318949 24759 fs_manager.cc:359] Metadata directory not provided
I20260928 06:28:25.318995 24759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:28:25.319093 24759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:28:25.320467 24759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/ts-1/data/instance:
uuid: "c439d4c662d84a308470ceba08705718"
format_stamp: "Formatted at 2026-09-28 06:28:25 on dist-test-slave-d59m"
server_key: "1f2a43a56a65eb6443be950c89f8bc6c"
server_key_iv: "509c9d04a08e29dbf6c9e6bdee9fb303"
server_key_version: "encryptionkey@0"
I20260928 06:28:25.321553 24759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/ts-1/wal/instance:
uuid: "c439d4c662d84a308470ceba08705718"
format_stamp: "Formatted at 2026-09-28 06:28:25 on dist-test-slave-d59m"
server_key: "1f2a43a56a65eb6443be950c89f8bc6c"
server_key_iv: "509c9d04a08e29dbf6c9e6bdee9fb303"
server_key_version: "encryptionkey@0"
I20260928 06:28:25.323872 24759 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:28:25.325410 24774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:25.325587 24759 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:28:25.325675 24759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/ts-1/wal
uuid: "c439d4c662d84a308470ceba08705718"
format_stamp: "Formatted at 2026-09-28 06:28:25 on dist-test-slave-d59m"
server_key: "1f2a43a56a65eb6443be950c89f8bc6c"
server_key_iv: "509c9d04a08e29dbf6c9e6bdee9fb303"
server_key_version: "encryptionkey@0"
I20260928 06:28:25.325729 24759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:28:25.341907 24759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:28:25.343927 24759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:28:25.344097 24759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:28:25.344702 24759 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:28:25.344744 24759 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:25.344769 24759 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:28:25.344780 24759 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 06:28:25 dist-test-slave-d59m krb5kdc[24502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576905, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.130@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:28:25.352485 24759 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.130:39959
I20260928 06:28:25.352568 24889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.130:39959 every 8 connection(s)
I20260928 06:28:25.352998 24759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
I20260928 06:28:25.356259 24596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:25.345194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:59587 (local address 127.22.225.190:46621)
0928 06:28:25.345269 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:25.345272 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:25.345427 (+ 155us) server_negotiation.cc:407] Connection header received
0928 06:28:25.345971 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:25.345973 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:25.345985 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:25.346044 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:25.346866 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:25.347896 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:25.349623 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:25.349982 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:25.350126 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:25.352881 (+ 2755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:25.352882 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:25.352882 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:25.352891 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:25.354763 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:25.355167 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:25.355168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:25.355168 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:25.355234 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:25.355485 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:25.355486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:25.355487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:25.355658 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:25.355730 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:25.356097 (+ 367us) server_negotiation.cc:299] Negotiation successful
0928 06:28:25.356188 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 06:28:25.356465 24783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:25.345146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:46621 (local address 127.22.225.130:59587)
0928 06:28:25.345305 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:25.345349 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:25.345847 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:25.346296 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:25.346306 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:25.346563 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:25.346757 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:25.346762 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:25.348010 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:25.348011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:25.349513 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:25.349515 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:25.350123 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:25.350124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:25.350294 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:25.350930 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:25.350940 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:25.352759 (+ 1819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:25.354857 (+ 2098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:25.354863 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:25.354864 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:25.355085 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:25.355304 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:25.355305 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:25.355305 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:25.355404 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:25.355745 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:25.355752 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:25.355921 (+ 169us) client_negotiation.cc:769] Sending connection context
0928 06:28:25.356036 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 06:28:25.356215 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":90,"threads_started":1}
I20260928 06:28:25.356971 24890 heartbeater.cc:347] Connected to a master server at 127.22.225.190:46621
I20260928 06:28:25.357079 24890 heartbeater.cc:464] Registering TS with master...
I20260928 06:28:25.357264 24890 heartbeater.cc:511] Master 127.22.225.190:46621 requested a full tablet report, sending...
I20260928 06:28:25.357630 24547 ts_manager.cc:194] Registered new tserver with Master: c439d4c662d84a308470ceba08705718 (127.22.225.130:39959)
I20260928 06:28:25.358078 24547 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.225.130@KRBTEST.COM'} at 127.22.225.130:59587
I20260928 06:28:25.360381 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 24759
I20260928 06:28:25.360484 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/ts-1/wal/instance
I20260928 06:28:25.360705 23430 external_mini_cluster.cc:1428] Setting key 3500698f404fc14e6994bf26a3d29646
I20260928 06:28:25.365401 24596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:25.358252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:55819 (local address 127.22.225.190:46621)
0928 06:28:25.358287 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:25.358289 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:25.358344 (+ 55us) server_negotiation.cc:407] Connection header received
0928 06:28:25.358445 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:25.358446 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:25.358457 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:25.358485 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:25.358854 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:25.359512 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:25.360612 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:25.360879 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:25.360978 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:25.362119 (+ 1141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:25.362121 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:25.362121 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:25.362128 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:25.363949 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:25.364405 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:25.364406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:25.364407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:25.364470 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:25.364710 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:25.364711 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:25.364712 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:25.364939 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:25.365019 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:25.365214 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 06:28:25.365301 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:28:25.365440 24896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:25.358097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:46621 (local address 127.22.225.130:55819)
0928 06:28:25.358274 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:25.358286 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:25.358374 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:25.358542 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:25.358543 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:25.358722 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:25.358770 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:25.358771 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:25.359624 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:25.359625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:25.360484 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:25.360487 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:25.360981 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:25.360982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:25.361064 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:25.361604 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:25.361610 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:25.362013 (+ 403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:25.364054 (+ 2041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:25.364056 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:25.364056 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:25.364299 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:25.364551 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:25.364552 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:25.364552 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:25.364648 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:25.365025 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:25.365033 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:25.365087 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 06:28:25.365173 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 06:28:25.365298 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":107,"threads_started":1}
WARNING: no policy specified for kudu/127.22.225.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab.
Entry for principal kudu/127.22.225.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab.
I20260928 06:28:25.388849 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.131:0
--local_ip_for_outbound_sockets=127.22.225.131
--webserver_interface=127.22.225.131
--webserver_port=0
--tserver_master_addrs=127.22.225.190:46621
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab
--principal=kudu/127.22.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:34263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:28:25.473100 24899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:28:25.473376 24899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:28:25.473419 24899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:28:25.474596 24899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:28:25.474660 24899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:28:25.474695 24899 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 06:28:25.474730 24899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.131
I20260928 06:28:25.476145 24899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:34263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab
--principal=kudu/127.22.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.225.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.225.190:46621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24899
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.131
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:28:25.476418 24899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:28:25.476706 24899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:28:25.481878 24904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:25.481880 24905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:25.481943 24907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:25.482184 24899 server_base.cc:1034] running on GCE node
I20260928 06:28:25.482374 24899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:28:25.482618 24899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:28:25.483786 24899 hybrid_clock.cc:648] HybridClock initialized: now 1790576905483772 us; error 27 us; skew 500 ppm
Sep 28 06:28:25 dist-test-slave-d59m krb5kdc[24502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576905, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:28:25.486850 24899 init.cc:377] Logged in from keytab as kudu/127.22.225.131@KRBTEST.COM (short username kudu)
I20260928 06:28:25.487354 24899 webserver.cc:460] Webserver started at http://127.22.225.131:34217/ using document root <none> and password file <none>
I20260928 06:28:25.487619 24899 fs_manager.cc:359] Metadata directory not provided
I20260928 06:28:25.487690 24899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:28:25.487847 24899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:28:25.488991 24899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/ts-2/data/instance:
uuid: "d2bbaa906b854eaeb416b9b294bda97f"
format_stamp: "Formatted at 2026-09-28 06:28:25 on dist-test-slave-d59m"
server_key: "3ed735a886456894f3ba4c6892470f8b"
server_key_iv: "2854d5c63075841e7e816762be1af31f"
server_key_version: "encryptionkey@0"
I20260928 06:28:25.489349 24899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/ts-2/wal/instance:
uuid: "d2bbaa906b854eaeb416b9b294bda97f"
format_stamp: "Formatted at 2026-09-28 06:28:25 on dist-test-slave-d59m"
server_key: "3ed735a886456894f3ba4c6892470f8b"
server_key_iv: "2854d5c63075841e7e816762be1af31f"
server_key_version: "encryptionkey@0"
I20260928 06:28:25.490717 24899 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:28:25.491506 24914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:25.491744 24899 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:28:25.491829 24899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/ts-2/wal
uuid: "d2bbaa906b854eaeb416b9b294bda97f"
format_stamp: "Formatted at 2026-09-28 06:28:25 on dist-test-slave-d59m"
server_key: "3ed735a886456894f3ba4c6892470f8b"
server_key_iv: "2854d5c63075841e7e816762be1af31f"
server_key_version: "encryptionkey@0"
I20260928 06:28:25.491896 24899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:28:25.505728 24899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:28:25.508311 24899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:28:25.508495 24899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:28:25.509289 24899 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:28:25.509343 24899 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:25.509380 24899 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:28:25.509411 24899 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 06:28:25 dist-test-slave-d59m krb5kdc[24502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576905, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.131@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:28:25.517597 24899 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.131:40979
I20260928 06:28:25.517704 25029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.131:40979 every 8 connection(s)
I20260928 06:28:25.518173 24899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
I20260928 06:28:25.521050 24596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:25.509643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:52891 (local address 127.22.225.190:46621)
0928 06:28:25.509690 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:25.509692 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:25.509979 (+ 287us) server_negotiation.cc:407] Connection header received
0928 06:28:25.510532 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:25.510534 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:25.510548 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:25.510614 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:25.511290 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:25.512107 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:25.513807 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:25.514116 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:25.514179 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:25.517639 (+ 3460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:25.517641 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:25.517642 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:25.517650 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:25.519504 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:25.519949 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:25.519951 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:25.519951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:25.520019 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:25.520288 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:25.520289 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:25.520289 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:25.520413 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:25.520486 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:25.520845 (+ 359us) server_negotiation.cc:299] Negotiation successful
0928 06:28:25.520950 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:28:25.521241 24923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:25.509660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:46621 (local address 127.22.225.131:52891)
0928 06:28:25.509864 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:25.509904 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:25.510412 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:25.510688 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:25.510692 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:25.510939 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:25.511190 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:25.511196 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:25.512202 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:25.512203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:25.513632 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:25.513635 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:25.514250 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:25.514252 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:25.514401 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:25.514967 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:25.514977 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:25.517511 (+ 2534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:25.519620 (+ 2109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:25.519627 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:25.519629 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:25.519856 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:25.520095 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:25.520096 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:25.520096 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:25.520211 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:25.520489 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:25.520496 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:25.520703 (+ 207us) client_negotiation.cc:769] Sending connection context
0928 06:28:25.520810 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 06:28:25.520987 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":146,"threads_started":1}
I20260928 06:28:25.521791 25030 heartbeater.cc:347] Connected to a master server at 127.22.225.190:46621
I20260928 06:28:25.521901 25030 heartbeater.cc:464] Registering TS with master...
I20260928 06:28:25.522131 25030 heartbeater.cc:511] Master 127.22.225.190:46621 requested a full tablet report, sending...
I20260928 06:28:25.522789 24546 ts_manager.cc:194] Registered new tserver with Master: d2bbaa906b854eaeb416b9b294bda97f (127.22.225.131:40979)
I20260928 06:28:25.523222 24546 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.225.131@KRBTEST.COM'} at 127.22.225.131:52891
I20260928 06:28:25.524044 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 24899
I20260928 06:28:25.524158 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/ts-2/wal/instance
I20260928 06:28:25.524351 23430 external_mini_cluster.cc:1428] Setting key 14fd1f82ac6f42bed9906642b86d25a1
I20260928 06:28:25.525333 23430 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:28:25.530943 24596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:25.523362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:60339 (local address 127.22.225.190:46621)
0928 06:28:25.523415 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:25.523417 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:25.523425 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:28:25.523443 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:25.523443 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:25.523452 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:25.523532 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:25.524571 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:25.525264 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:25.526375 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:25.526705 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:25.526750 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:25.527795 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:25.527797 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:25.527797 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:25.527805 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:25.529541 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:25.529988 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:25.529990 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:25.529990 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:25.530056 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:25.530313 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:25.530314 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:25.530315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:25.530466 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:25.530546 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:25.530752 (+ 206us) server_negotiation.cc:299] Negotiation successful
0928 06:28:25.530837 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 06:28:25.531015 25036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:25.522815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:46621 (local address 127.22.225.131:60339)
0928 06:28:25.522986 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:25.523001 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:25.523082 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:25.523538 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:25.523539 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:25.523746 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:25.523813 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:25.523815 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:25.525431 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:25.525432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:25.526268 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:25.526271 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:25.526828 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:25.526829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:25.526911 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:25.527412 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:25.527416 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:25.527702 (+ 286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:25.529634 (+ 1932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:25.529635 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:25.529636 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:25.529899 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:25.530149 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:25.530150 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:25.530151 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:25.530234 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:25.530549 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:25.530551 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:25.530610 (+ 59us) client_negotiation.cc:769] Sending connection context
0928 06:28:25.530698 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 06:28:25.530844 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":77,"threads_started":1}
Sep 28 06:28:25 dist-test-slave-d59m krb5kdc[24502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576905, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 06:28:25 dist-test-slave-d59m krb5kdc[24502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576905, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:28:25.552928 24596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:25.544506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37820 (local address 127.22.225.190:46621)
0928 06:28:25.544556 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:25.544558 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:25.544668 (+ 110us) server_negotiation.cc:407] Connection header received
0928 06:28:25.544749 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:25.544750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:25.544760 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:25.544811 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:25.545552 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:25.546041 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:25.547128 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:25.547362 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:25.547419 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:25.550365 (+ 2946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:25.550367 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:25.550367 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:25.550374 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:25.551754 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:25.552182 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:25.552183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:25.552183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:25.552233 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:25.552434 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:25.552435 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:25.552435 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:25.552537 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:25.552671 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:25.552797 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 06:28:25.552859 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 06:28:25.554847 24546 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37820:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:28:25.556469 24546 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:28:25.563844 25047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:25.559039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.131:40979 (local address 127.0.0.1:59810)
0928 06:28:25.559155 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:25.559196 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:25.559266 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:25.559653 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:25.559656 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:25.559659 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:28:25.559747 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:25.559749 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:25.561155 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:25.561156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:25.562689 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:25.562691 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:25.563586 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:25.563587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:25.563638 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:25.563641 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:28:25.563671 (+ 30us) client_negotiation.cc:240] Negotiation successful
0928 06:28:25.563751 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"mutex_wait_us":371,"thread_start_us":85,"threads_started":1}
I20260928 06:28:25.563951 25050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:25.559293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59810 (local address 127.22.225.131:40979)
0928 06:28:25.559458 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:25.559464 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:25.559474 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:28:25.559517 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:25.559519 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:25.559540 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:25.559822 (+ 282us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:28:25.559887 (+ 65us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:25.561053 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:25.562775 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:25.563499 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:25.563735 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:25.563789 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:25.563815 (+ 26us) server_negotiation.cc:299] Negotiation successful
0928 06:28:25.563850 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":83,"threads_started":1}
I20260928 06:28:25.564402 25048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:25.559044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.130:39959 (local address 127.0.0.1:57648)
0928 06:28:25.559516 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:25.559525 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:25.559591 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:25.559871 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:25.559872 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:25.559873 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:28:25.559926 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:25.559927 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:25.561288 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:25.561289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:25.562895 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:25.562898 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:25.564164 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:25.564165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:25.564226 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:25.564229 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:28:25.564272 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 06:28:25.564307 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"mutex_wait_us":313,"thread_start_us":61,"threads_started":1}
I20260928 06:28:25.564661 25049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:25.559046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.129:43685 (local address 127.0.0.1:49218)
0928 06:28:25.559670 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:25.559678 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:25.559733 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:25.560010 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:25.560011 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:25.560012 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:28:25.560060 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:25.560062 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:25.561291 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:25.561291 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:25.562462 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:25.562465 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:25.563492 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:25.563493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:25.563576 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:25.563604 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 06:28:25.563671 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 06:28:25.564285 (+ 614us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":86,"threads_started":1}
I20260928 06:28:25.564720 25052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:25.559307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57648 (local address 127.22.225.130:39959)
0928 06:28:25.559499 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:25.559521 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:25.559645 (+ 124us) server_negotiation.cc:407] Connection header received
0928 06:28:25.559691 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:25.559693 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:25.559722 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:25.559800 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:28:25.560017 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:25.561194 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:25.562986 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:25.564013 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:25.564080 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:25.564135 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:25.564553 (+ 418us) server_negotiation.cc:299] Negotiation successful
0928 06:28:25.564624 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":89,"threads_started":1}
I20260928 06:28:25.564643 24964 tablet_service.cc:1467] Processing CreateTablet for tablet b91a57c9f0e344f98d893fcbda78fa32 (DEFAULT_TABLE table=test-table [id=0db79e07aeba45edaf64b7b0367117c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:28:25.565075 24964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b91a57c9f0e344f98d893fcbda78fa32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:25.565344 25051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:25.559303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49218 (local address 127.22.225.129:43685)
0928 06:28:25.559621 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:25.559625 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:25.559792 (+ 167us) server_negotiation.cc:407] Connection header received
0928 06:28:25.559837 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:25.559839 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:25.559854 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:25.559926 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:28:25.560139 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:25.561209 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:25.562575 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:25.563325 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:25.563398 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:25.563454 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:25.564372 (+ 918us) server_negotiation.cc:299] Negotiation successful
0928 06:28:25.564414 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":74,"threads_started":1}
I20260928 06:28:25.565407 24824 tablet_service.cc:1467] Processing CreateTablet for tablet b91a57c9f0e344f98d893fcbda78fa32 (DEFAULT_TABLE table=test-table [id=0db79e07aeba45edaf64b7b0367117c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:28:25.565743 24824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b91a57c9f0e344f98d893fcbda78fa32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:25.567971 25053 tablet_bootstrap.cc:492] T b91a57c9f0e344f98d893fcbda78fa32 P d2bbaa906b854eaeb416b9b294bda97f: Bootstrap starting.
I20260928 06:28:25.568857 25053 tablet_bootstrap.cc:654] T b91a57c9f0e344f98d893fcbda78fa32 P d2bbaa906b854eaeb416b9b294bda97f: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:25.569213 25053 log.cc:824] T b91a57c9f0e344f98d893fcbda78fa32 P d2bbaa906b854eaeb416b9b294bda97f: Log is configured to *not* fsync() on all Append() calls
I20260928 06:28:25.569245 25054 tablet_bootstrap.cc:492] T b91a57c9f0e344f98d893fcbda78fa32 P c439d4c662d84a308470ceba08705718: Bootstrap starting.
I20260928 06:28:25.569248 24684 tablet_service.cc:1467] Processing CreateTablet for tablet b91a57c9f0e344f98d893fcbda78fa32 (DEFAULT_TABLE table=test-table [id=0db79e07aeba45edaf64b7b0367117c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:28:25.569602 24684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b91a57c9f0e344f98d893fcbda78fa32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:25.569941 25054 tablet_bootstrap.cc:654] T b91a57c9f0e344f98d893fcbda78fa32 P c439d4c662d84a308470ceba08705718: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:25.569962 25053 tablet_bootstrap.cc:492] T b91a57c9f0e344f98d893fcbda78fa32 P d2bbaa906b854eaeb416b9b294bda97f: No bootstrap required, opened a new log
I20260928 06:28:25.570022 25053 ts_tablet_manager.cc:1397] T b91a57c9f0e344f98d893fcbda78fa32 P d2bbaa906b854eaeb416b9b294bda97f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:28:25.570314 25054 log.cc:824] T b91a57c9f0e344f98d893fcbda78fa32 P c439d4c662d84a308470ceba08705718: Log is configured to *not* fsync() on all Append() calls
I20260928 06:28:25.570966 25054 tablet_bootstrap.cc:492] T b91a57c9f0e344f98d893fcbda78fa32 P c439d4c662d84a308470ceba08705718: No bootstrap required, opened a new log
I20260928 06:28:25.571043 25054 ts_tablet_manager.cc:1397] T b91a57c9f0e344f98d893fcbda78fa32 P c439d4c662d84a308470ceba08705718: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:28:25.572094 25053 raft_consensus.cc:348] T b91a57c9f0e344f98d893fcbda78fa32 P d2bbaa906b854eaeb416b9b294bda97f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59eeadf53cb043659a02cba07a4300ef" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 43685 } } peers { permanent_uuid: "d2bbaa906b854eaeb416b9b294bda97f" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40979 } } peers { permanent_uuid: "c439d4c662d84a308470ceba08705718" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 39959 } }
I20260928 06:28:25.572286 25053 raft_consensus.cc:374] T b91a57c9f0e344f98d893fcbda78fa32 P d2bbaa906b854eaeb416b9b294bda97f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:25.572329 25053 raft_consensus.cc:729] T b91a57c9f0e344f98d893fcbda78fa32 P d2bbaa906b854eaeb416b9b294bda97f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2bbaa906b854eaeb416b9b294bda97f, State: Initialized, Role: FOLLOWER
I20260928 06:28:25.572320 25054 raft_consensus.cc:348] T b91a57c9f0e344f98d893fcbda78fa32 P c439d4c662d84a308470ceba08705718 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59eeadf53cb043659a02cba07a4300ef" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 43685 } } peers { permanent_uuid: "d2bbaa906b854eaeb416b9b294bda97f" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40979 } } peers { permanent_uuid: "c439d4c662d84a308470ceba08705718" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 39959 } }
I20260928 06:28:25.572469 25053 consensus_queue.cc:260] T b91a57c9f0e344f98d893fcbda78fa32 P d2bbaa906b854eaeb416b9b294bda97f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59eeadf53cb043659a02cba07a4300ef" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 43685 } } peers { permanent_uuid: "d2bbaa906b854eaeb416b9b294bda97f" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40979 } } peers { permanent_uuid: "c439d4c662d84a308470ceba08705718" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 39959 } }
I20260928 06:28:25.572535 25054 raft_consensus.cc:374] T b91a57c9f0e344f98d893fcbda78fa32 P c439d4c662d84a308470ceba08705718 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:25.572582 25054 raft_consensus.cc:729] T b91a57c9f0e344f98d893fcbda78fa32 P c439d4c662d84a308470ceba08705718 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c439d4c662d84a308470ceba08705718, State: Initialized, Role: FOLLOWER
I20260928 06:28:25.572741 25053 ts_tablet_manager.cc:1428] T b91a57c9f0e344f98d893fcbda78fa32 P d2bbaa906b854eaeb416b9b294bda97f: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260928 06:28:25.572722 25054 consensus_queue.cc:260] T b91a57c9f0e344f98d893fcbda78fa32 P c439d4c662d84a308470ceba08705718 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59eeadf53cb043659a02cba07a4300ef" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 43685 } } peers { permanent_uuid: "d2bbaa906b854eaeb416b9b294bda97f" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40979 } } peers { permanent_uuid: "c439d4c662d84a308470ceba08705718" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 39959 } }
I20260928 06:28:25.572891 25030 heartbeater.cc:503] Master 127.22.225.190:46621 was elected leader, sending a full tablet report...
I20260928 06:28:25.572975 25054 ts_tablet_manager.cc:1428] T b91a57c9f0e344f98d893fcbda78fa32 P c439d4c662d84a308470ceba08705718: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 06:28:25.573069 24890 heartbeater.cc:503] Master 127.22.225.190:46621 was elected leader, sending a full tablet report...
I20260928 06:28:25.573276 25059 tablet_bootstrap.cc:492] T b91a57c9f0e344f98d893fcbda78fa32 P 59eeadf53cb043659a02cba07a4300ef: Bootstrap starting.
I20260928 06:28:25.573827 25059 tablet_bootstrap.cc:654] T b91a57c9f0e344f98d893fcbda78fa32 P 59eeadf53cb043659a02cba07a4300ef: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:25.574115 25059 log.cc:824] T b91a57c9f0e344f98d893fcbda78fa32 P 59eeadf53cb043659a02cba07a4300ef: Log is configured to *not* fsync() on all Append() calls
I20260928 06:28:25.574757 25059 tablet_bootstrap.cc:492] T b91a57c9f0e344f98d893fcbda78fa32 P 59eeadf53cb043659a02cba07a4300ef: No bootstrap required, opened a new log
I20260928 06:28:25.574819 25059 ts_tablet_manager.cc:1397] T b91a57c9f0e344f98d893fcbda78fa32 P 59eeadf53cb043659a02cba07a4300ef: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:28:25.576535 25059 raft_consensus.cc:348] T b91a57c9f0e344f98d893fcbda78fa32 P 59eeadf53cb043659a02cba07a4300ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59eeadf53cb043659a02cba07a4300ef" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 43685 } } peers { permanent_uuid: "d2bbaa906b854eaeb416b9b294bda97f" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40979 } } peers { permanent_uuid: "c439d4c662d84a308470ceba08705718" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 39959 } }
I20260928 06:28:25.576710 25059 raft_consensus.cc:374] T b91a57c9f0e344f98d893fcbda78fa32 P 59eeadf53cb043659a02cba07a4300ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:25.576741 25059 raft_consensus.cc:729] T b91a57c9f0e344f98d893fcbda78fa32 P 59eeadf53cb043659a02cba07a4300ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59eeadf53cb043659a02cba07a4300ef, State: Initialized, Role: FOLLOWER
I20260928 06:28:25.576843 25059 consensus_queue.cc:260] T b91a57c9f0e344f98d893fcbda78fa32 P 59eeadf53cb043659a02cba07a4300ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59eeadf53cb043659a02cba07a4300ef" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 43685 } } peers { permanent_uuid: "d2bbaa906b854eaeb416b9b294bda97f" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40979 } } peers { permanent_uuid: "c439d4c662d84a308470ceba08705718" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 39959 } }
I20260928 06:28:25.577072 25059 ts_tablet_manager.cc:1428] T b91a57c9f0e344f98d893fcbda78fa32 P 59eeadf53cb043659a02cba07a4300ef: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 06:28:25.577121 24750 heartbeater.cc:503] Master 127.22.225.190:46621 was elected leader, sending a full tablet report...
W20260928 06:28:25.603565 24891 tablet.cc:2369] T b91a57c9f0e344f98d893fcbda78fa32 P c439d4c662d84a308470ceba08705718: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:28:25.683430 24751 tablet.cc:2369] T b91a57c9f0e344f98d893fcbda78fa32 P 59eeadf53cb043659a02cba07a4300ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:28:25.768901 25031 tablet.cc:2369] T b91a57c9f0e344f98d893fcbda78fa32 P d2bbaa906b854eaeb416b9b294bda97f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:28:25.784509 25058 raft_consensus.cc:482] T b91a57c9f0e344f98d893fcbda78fa32 P c439d4c662d84a308470ceba08705718 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:28:25.784687 25058 raft_consensus.cc:504] T b91a57c9f0e344f98d893fcbda78fa32 P c439d4c662d84a308470ceba08705718 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59eeadf53cb043659a02cba07a4300ef" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 43685 } } peers { permanent_uuid: "d2bbaa906b854eaeb416b9b294bda97f" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40979 } } peers { permanent_uuid: "c439d4c662d84a308470ceba08705718" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 39959 } }
I20260928 06:28:25.785120 25058 leader_election.cc:288] T b91a57c9f0e344f98d893fcbda78fa32 P c439d4c662d84a308470ceba08705718 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59eeadf53cb043659a02cba07a4300ef (127.22.225.129:43685), d2bbaa906b854eaeb416b9b294bda97f (127.22.225.131:40979)
I20260928 06:28:25.786769 25057 raft_consensus.cc:482] T b91a57c9f0e344f98d893fcbda78fa32 P d2bbaa906b854eaeb416b9b294bda97f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:28:25.786875 25057 raft_consensus.cc:504] T b91a57c9f0e344f98d893fcbda78fa32 P d2bbaa906b854eaeb416b9b294bda97f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59eeadf53cb043659a02cba07a4300ef" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 43685 } } peers { permanent_uuid: "d2bbaa906b854eaeb416b9b294bda97f" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40979 } } peers { permanent_uuid: "c439d4c662d84a308470ceba08705718" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 39959 } }
I20260928 06:28:25.787199 25057 leader_election.cc:288] T b91a57c9f0e344f98d893fcbda78fa32 P d2bbaa906b854eaeb416b9b294bda97f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59eeadf53cb043659a02cba07a4300ef (127.22.225.129:43685), c439d4c662d84a308470ceba08705718 (127.22.225.130:39959)
I20260928 06:28:25.788965 24783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:25.785290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.131:40979 (local address 127.22.225.130:33149)
0928 06:28:25.785371 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:25.785385 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:25.785441 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:25.785632 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:25.785634 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:25.785637 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:28:25.785714 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:25.785716 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:25.786690 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:25.786691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:25.787881 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:25.787884 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:25.788710 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:25.788711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:25.788795 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:25.788800 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:28:25.788850 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 06:28:25.788890 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 06:28:25.789549 25050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:25.785450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:33149 (local address 127.22.225.131:40979)
0928 06:28:25.785496 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:25.785498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:25.785508 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:28:25.785547 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:25.785548 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:25.785559 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:25.785624 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:28:25.785781 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:25.786617 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:25.787980 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:25.788562 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:25.788661 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:25.788673 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:25.789356 (+ 683us) server_negotiation.cc:299] Negotiation successful
0928 06:28:25.789476 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":73,"server-negotiator.queue_time_us":23}
I20260928 06:28:25.790836 24984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91a57c9f0e344f98d893fcbda78fa32" candidate_uuid: "c439d4c662d84a308470ceba08705718" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2bbaa906b854eaeb416b9b294bda97f" is_pre_election: true
I20260928 06:28:25.790958 24984 raft_consensus.cc:2454] T b91a57c9f0e344f98d893fcbda78fa32 P d2bbaa906b854eaeb416b9b294bda97f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c439d4c662d84a308470ceba08705718 in term 0.
I20260928 06:28:25.790998 25051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:25.785448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:36063 (local address 127.22.225.129:43685)
0928 06:28:25.785496 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:25.785498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:25.785725 (+ 227us) server_negotiation.cc:407] Connection header received
0928 06:28:25.785826 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:25.785827 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:25.785838 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:25.785906 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:28:25.786108 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:25.786878 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:25.790103 (+ 3225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:25.790637 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:25.790698 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:25.790706 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:25.790888 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 06:28:25.790927 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 06:28:25.791220 24778 leader_election.cc:302] T b91a57c9f0e344f98d893fcbda78fa32 P c439d4c662d84a308470ceba08705718 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c439d4c662d84a308470ceba08705718, d2bbaa906b854eaeb416b9b294bda97f; no voters:
I20260928 06:28:25.791415 25052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:25.787817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:49105 (local address 127.22.225.130:39959)
0928 06:28:25.787856 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:25.787858 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:25.788211 (+ 353us) server_negotiation.cc:407] Connection header received
0928 06:28:25.788296 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:25.788296 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:25.788305 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:25.788334 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:28:25.788518 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:25.789216 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:25.790487 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:25.790974 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:25.791013 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:25.791020 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:25.791290 (+ 270us) server_negotiation.cc:299] Negotiation successful
0928 06:28:25.791326 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 06:28:25.791682 25063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:25.787646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.130:39959 (local address 127.22.225.131:49105)
0928 06:28:25.788138 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:25.788146 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:25.788219 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:25.788392 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:25.788393 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:25.788394 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:28:25.788449 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:25.788450 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:25.789299 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:25.789300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:25.790389 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:25.790392 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:25.791139 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:25.791140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:25.791206 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:25.791209 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:28:25.791542 (+ 333us) client_negotiation.cc:240] Negotiation successful
0928 06:28:25.791583 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":86,"threads_started":1}
I20260928 06:28:25.791841 25058 raft_consensus.cc:2781] T b91a57c9f0e344f98d893fcbda78fa32 P c439d4c662d84a308470ceba08705718 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:28:25.791975 24923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:25.787389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.129:43685 (local address 127.22.225.131:37651)
0928 06:28:25.787432 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:25.787440 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:25.787472 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:25.788246 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:25.788247 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:25.788249 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:28:25.788304 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:25.788305 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:25.789258 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:25.789259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:25.790450 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:25.790453 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:25.791739 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:25.791740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:25.791828 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:25.791831 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:28:25.791864 (+ 33us) client_negotiation.cc:240] Negotiation successful
0928 06:28:25.791907 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":34}
I20260928 06:28:25.791939 24844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91a57c9f0e344f98d893fcbda78fa32" candidate_uuid: "d2bbaa906b854eaeb416b9b294bda97f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c439d4c662d84a308470ceba08705718" is_pre_election: true
I20260928 06:28:25.791941 25058 raft_consensus.cc:482] T b91a57c9f0e344f98d893fcbda78fa32 P c439d4c662d84a308470ceba08705718 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:28:25.792282 25058 raft_consensus.cc:3037] T b91a57c9f0e344f98d893fcbda78fa32 P c439d4c662d84a308470ceba08705718 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:25.792308 25064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:25.787994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:37651 (local address 127.22.225.129:43685)
0928 06:28:25.788122 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:25.788124 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:25.788132 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:28:25.788154 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:25.788155 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:25.788165 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:25.788236 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:28:25.788386 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:25.789160 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:25.791054 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:25.791634 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:25.792177 (+ 543us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:25.792186 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:25.792205 (+ 19us) server_negotiation.cc:299] Negotiation successful
0928 06:28:25.792246 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":74,"threads_started":1}
I20260928 06:28:25.792632 24704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91a57c9f0e344f98d893fcbda78fa32" candidate_uuid: "d2bbaa906b854eaeb416b9b294bda97f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59eeadf53cb043659a02cba07a4300ef" is_pre_election: true
I20260928 06:28:25.792776 24704 raft_consensus.cc:2454] T b91a57c9f0e344f98d893fcbda78fa32 P 59eeadf53cb043659a02cba07a4300ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2bbaa906b854eaeb416b9b294bda97f in term 0.
I20260928 06:28:25.792243 25062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:25.785290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.129:43685 (local address 127.22.225.130:36063)
0928 06:28:25.785673 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:25.785682 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:25.785736 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:25.785922 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:25.785923 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:25.785925 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:28:25.786003 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:25.786004 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:25.786998 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:25.786999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:25.790010 (+ 3011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:25.790013 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:25.790749 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:25.790750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:25.790813 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:25.790816 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:28:25.791952 (+ 1136us) client_negotiation.cc:240] Negotiation successful
0928 06:28:25.792026 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"mutex_wait_us":236,"thread_start_us":81,"threads_started":1}
I20260928 06:28:25.792991 24917 leader_election.cc:302] T b91a57c9f0e344f98d893fcbda78fa32 P d2bbaa906b854eaeb416b9b294bda97f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59eeadf53cb043659a02cba07a4300ef, d2bbaa906b854eaeb416b9b294bda97f; no voters:
I20260928 06:28:25.793081 24704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91a57c9f0e344f98d893fcbda78fa32" candidate_uuid: "c439d4c662d84a308470ceba08705718" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59eeadf53cb043659a02cba07a4300ef" is_pre_election: true
I20260928 06:28:25.793143 24704 raft_consensus.cc:2454] T b91a57c9f0e344f98d893fcbda78fa32 P 59eeadf53cb043659a02cba07a4300ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c439d4c662d84a308470ceba08705718 in term 0.
I20260928 06:28:25.793219 25057 raft_consensus.cc:2781] T b91a57c9f0e344f98d893fcbda78fa32 P d2bbaa906b854eaeb416b9b294bda97f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:28:25.793277 25057 raft_consensus.cc:482] T b91a57c9f0e344f98d893fcbda78fa32 P d2bbaa906b854eaeb416b9b294bda97f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:28:25.793304 25057 raft_consensus.cc:3037] T b91a57c9f0e344f98d893fcbda78fa32 P d2bbaa906b854eaeb416b9b294bda97f [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:25.793740 25058 raft_consensus.cc:504] T b91a57c9f0e344f98d893fcbda78fa32 P c439d4c662d84a308470ceba08705718 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59eeadf53cb043659a02cba07a4300ef" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 43685 } } peers { permanent_uuid: "d2bbaa906b854eaeb416b9b294bda97f" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40979 } } peers { permanent_uuid: "c439d4c662d84a308470ceba08705718" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 39959 } }
I20260928 06:28:25.793892 24844 raft_consensus.cc:2379] T b91a57c9f0e344f98d893fcbda78fa32 P c439d4c662d84a308470ceba08705718 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d2bbaa906b854eaeb416b9b294bda97f in current term 1: Already voted for candidate c439d4c662d84a308470ceba08705718 in this term.
I20260928 06:28:25.794193 25057 raft_consensus.cc:504] T b91a57c9f0e344f98d893fcbda78fa32 P d2bbaa906b854eaeb416b9b294bda97f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59eeadf53cb043659a02cba07a4300ef" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 43685 } } peers { permanent_uuid: "d2bbaa906b854eaeb416b9b294bda97f" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40979 } } peers { permanent_uuid: "c439d4c662d84a308470ceba08705718" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 39959 } }
I20260928 06:28:25.794324 25057 leader_election.cc:288] T b91a57c9f0e344f98d893fcbda78fa32 P d2bbaa906b854eaeb416b9b294bda97f [CANDIDATE]: Term 1 election: Requested vote from peers 59eeadf53cb043659a02cba07a4300ef (127.22.225.129:43685), c439d4c662d84a308470ceba08705718 (127.22.225.130:39959)
I20260928 06:28:25.794565 24704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91a57c9f0e344f98d893fcbda78fa32" candidate_uuid: "d2bbaa906b854eaeb416b9b294bda97f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59eeadf53cb043659a02cba07a4300ef"
I20260928 06:28:25.794641 24704 raft_consensus.cc:3037] T b91a57c9f0e344f98d893fcbda78fa32 P 59eeadf53cb043659a02cba07a4300ef [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:25.794660 24703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91a57c9f0e344f98d893fcbda78fa32" candidate_uuid: "c439d4c662d84a308470ceba08705718" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59eeadf53cb043659a02cba07a4300ef"
I20260928 06:28:25.794901 24844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91a57c9f0e344f98d893fcbda78fa32" candidate_uuid: "d2bbaa906b854eaeb416b9b294bda97f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c439d4c662d84a308470ceba08705718"
I20260928 06:28:25.794967 24844 raft_consensus.cc:2379] T b91a57c9f0e344f98d893fcbda78fa32 P c439d4c662d84a308470ceba08705718 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d2bbaa906b854eaeb416b9b294bda97f in current term 1: Already voted for candidate c439d4c662d84a308470ceba08705718 in this term.
I20260928 06:28:25.795169 25058 leader_election.cc:288] T b91a57c9f0e344f98d893fcbda78fa32 P c439d4c662d84a308470ceba08705718 [CANDIDATE]: Term 1 election: Requested vote from peers 59eeadf53cb043659a02cba07a4300ef (127.22.225.129:43685), d2bbaa906b854eaeb416b9b294bda97f (127.22.225.131:40979)
I20260928 06:28:25.795181 24984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91a57c9f0e344f98d893fcbda78fa32" candidate_uuid: "c439d4c662d84a308470ceba08705718" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2bbaa906b854eaeb416b9b294bda97f"
I20260928 06:28:25.795262 24984 raft_consensus.cc:2379] T b91a57c9f0e344f98d893fcbda78fa32 P d2bbaa906b854eaeb416b9b294bda97f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c439d4c662d84a308470ceba08705718 in current term 1: Already voted for candidate d2bbaa906b854eaeb416b9b294bda97f in this term.
I20260928 06:28:25.795431 24778 leader_election.cc:302] T b91a57c9f0e344f98d893fcbda78fa32 P c439d4c662d84a308470ceba08705718 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c439d4c662d84a308470ceba08705718; no voters: 59eeadf53cb043659a02cba07a4300ef, d2bbaa906b854eaeb416b9b294bda97f
I20260928 06:28:25.795516 25058 raft_consensus.cc:2726] T b91a57c9f0e344f98d893fcbda78fa32 P c439d4c662d84a308470ceba08705718 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260928 06:28:25.795532 24704 raft_consensus.cc:2454] T b91a57c9f0e344f98d893fcbda78fa32 P 59eeadf53cb043659a02cba07a4300ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2bbaa906b854eaeb416b9b294bda97f in term 1.
I20260928 06:28:25.795660 24917 leader_election.cc:302] T b91a57c9f0e344f98d893fcbda78fa32 P d2bbaa906b854eaeb416b9b294bda97f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59eeadf53cb043659a02cba07a4300ef, d2bbaa906b854eaeb416b9b294bda97f; no voters: c439d4c662d84a308470ceba08705718
I20260928 06:28:25.795750 25057 raft_consensus.cc:2781] T b91a57c9f0e344f98d893fcbda78fa32 P d2bbaa906b854eaeb416b9b294bda97f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:28:25.795892 25057 raft_consensus.cc:686] T b91a57c9f0e344f98d893fcbda78fa32 P d2bbaa906b854eaeb416b9b294bda97f [term 1 LEADER]: Becoming Leader. State: Replica: d2bbaa906b854eaeb416b9b294bda97f, State: Running, Role: LEADER
I20260928 06:28:25.795998 25057 consensus_queue.cc:237] T b91a57c9f0e344f98d893fcbda78fa32 P d2bbaa906b854eaeb416b9b294bda97f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59eeadf53cb043659a02cba07a4300ef" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 43685 } } peers { permanent_uuid: "d2bbaa906b854eaeb416b9b294bda97f" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40979 } } peers { permanent_uuid: "c439d4c662d84a308470ceba08705718" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 39959 } }
I20260928 06:28:25.796779 24547 catalog_manager.cc:5495] T b91a57c9f0e344f98d893fcbda78fa32 P d2bbaa906b854eaeb416b9b294bda97f reported cstate change: term changed from 0 to 1, leader changed from <none> to d2bbaa906b854eaeb416b9b294bda97f (127.22.225.131). New cstate: current_term: 1 leader_uuid: "d2bbaa906b854eaeb416b9b294bda97f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59eeadf53cb043659a02cba07a4300ef" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 43685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2bbaa906b854eaeb416b9b294bda97f" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c439d4c662d84a308470ceba08705718" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 39959 } health_report { overall_health: UNKNOWN } } }
I20260928 06:28:25.846051 25050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:25.843370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59816 (local address 127.22.225.131:40979)
0928 06:28:25.843441 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:25.843443 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:25.843456 (+ 13us) server_negotiation.cc:407] Connection header received
0928 06:28:25.843586 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:25.843588 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:25.843610 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:25.843697 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:28:25.843845 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:25.844335 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:25.845188 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:25.845374 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:25.845457 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:25.845655 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:28:25.845761 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:28:25.845852 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:25.845925 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 06:28:25.845945 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 06:28:25.847877 24844 raft_consensus.cc:1261] T b91a57c9f0e344f98d893fcbda78fa32 P c439d4c662d84a308470ceba08705718 [term 1 FOLLOWER]: Refusing update from remote peer d2bbaa906b854eaeb416b9b294bda97f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:28:25.847874 24704 raft_consensus.cc:1261] T b91a57c9f0e344f98d893fcbda78fa32 P 59eeadf53cb043659a02cba07a4300ef [term 1 FOLLOWER]: Refusing update from remote peer d2bbaa906b854eaeb416b9b294bda97f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:28:25.848110 25057 consensus_queue.cc:1035] T b91a57c9f0e344f98d893fcbda78fa32 P d2bbaa906b854eaeb416b9b294bda97f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c439d4c662d84a308470ceba08705718" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 39959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:28:25.848264 25065 consensus_queue.cc:1035] T b91a57c9f0e344f98d893fcbda78fa32 P d2bbaa906b854eaeb416b9b294bda97f [LEADER]: Connected to new peer: Peer: permanent_uuid: "59eeadf53cb043659a02cba07a4300ef" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 43685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:28:25.852006 25071 mvcc.cc:205] Tried to move back new op lower bound from 7334203006350196736 to 7334203006140907520. Current Snapshot: MvccSnapshot[applied={T|T < 7334203006350196736}]
I20260928 06:28:25.852658 25069 mvcc.cc:205] Tried to move back new op lower bound from 7334203006350196736 to 7334203006140907520. Current Snapshot: MvccSnapshot[applied={T|T < 7334203006350196736 or (T in {7334203006350196736})}]
I20260928 06:28:25.852936 25068 mvcc.cc:205] Tried to move back new op lower bound from 7334203006350196736 to 7334203006140907520. Current Snapshot: MvccSnapshot[applied={T|T < 7334203006350196736 or (T in {7334203006350196736})}]
I20260928 06:28:25.854472 24547 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37820:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:28:25.854568 24547 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37820:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:28:25.855283 24547 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4918f18b857f4b0696388f71b4355706: Sending DeleteTablet for 3 replicas of tablet b91a57c9f0e344f98d893fcbda78fa32
I20260928 06:28:25.855669 24824 tablet_service.cc:1514] Processing DeleteTablet for tablet b91a57c9f0e344f98d893fcbda78fa32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:28:25 UTC) from {username='kudu', principal='kudu/127.22.225.190@KRBTEST.COM'} at 127.0.0.1:57648
I20260928 06:28:25.855682 24964 tablet_service.cc:1514] Processing DeleteTablet for tablet b91a57c9f0e344f98d893fcbda78fa32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:28:25 UTC) from {username='kudu', principal='kudu/127.22.225.190@KRBTEST.COM'} at 127.0.0.1:59810
I20260928 06:28:25.855698 24684 tablet_service.cc:1514] Processing DeleteTablet for tablet b91a57c9f0e344f98d893fcbda78fa32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:28:25 UTC) from {username='kudu', principal='kudu/127.22.225.190@KRBTEST.COM'} at 127.0.0.1:49218
I20260928 06:28:25.856107 25079 tablet_replica.cc:330] T b91a57c9f0e344f98d893fcbda78fa32 P c439d4c662d84a308470ceba08705718: stopping tablet replica
I20260928 06:28:25.856212 25079 raft_consensus.cc:2229] T b91a57c9f0e344f98d893fcbda78fa32 P c439d4c662d84a308470ceba08705718 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:28:25.856168 24547 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37820:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:28:25.856294 24547 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:28:25.856310 25079 raft_consensus.cc:2258] T b91a57c9f0e344f98d893fcbda78fa32 P c439d4c662d84a308470ceba08705718 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:28:25.856767 25077 tablet_replica.cc:330] T b91a57c9f0e344f98d893fcbda78fa32 P d2bbaa906b854eaeb416b9b294bda97f: stopping tablet replica
I20260928 06:28:25.856917 25077 raft_consensus.cc:2229] T b91a57c9f0e344f98d893fcbda78fa32 P d2bbaa906b854eaeb416b9b294bda97f [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:28:25.857025 25077 raft_consensus.cc:2258] T b91a57c9f0e344f98d893fcbda78fa32 P d2bbaa906b854eaeb416b9b294bda97f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:28:25.857177 25079 ts_tablet_manager.cc:1905] T b91a57c9f0e344f98d893fcbda78fa32 P c439d4c662d84a308470ceba08705718: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:28:25.857524 24684 tablet_service.cc:1467] Processing CreateTablet for tablet 7c1e783f0fa44ab98db665a30353573e (DEFAULT_TABLE table=test-table [id=d20ac0b5aad947099555f2c0f674e503]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:28:25.857568 24824 tablet_service.cc:1467] Processing CreateTablet for tablet 7c1e783f0fa44ab98db665a30353573e (DEFAULT_TABLE table=test-table [id=d20ac0b5aad947099555f2c0f674e503]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:28:25.857664 24684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c1e783f0fa44ab98db665a30353573e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:25.857674 24824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c1e783f0fa44ab98db665a30353573e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:25.859094 25059 tablet_bootstrap.cc:492] T 7c1e783f0fa44ab98db665a30353573e P 59eeadf53cb043659a02cba07a4300ef: Bootstrap starting.
I20260928 06:28:25.859066 25054 tablet_bootstrap.cc:492] T 7c1e783f0fa44ab98db665a30353573e P c439d4c662d84a308470ceba08705718: Bootstrap starting.
I20260928 06:28:25.859241 24964 tablet_service.cc:1467] Processing CreateTablet for tablet 7c1e783f0fa44ab98db665a30353573e (DEFAULT_TABLE table=test-table [id=d20ac0b5aad947099555f2c0f674e503]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:28:25.859368 24964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c1e783f0fa44ab98db665a30353573e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:25.859519 25054 tablet_bootstrap.cc:654] T 7c1e783f0fa44ab98db665a30353573e P c439d4c662d84a308470ceba08705718: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:25.859547 25059 tablet_bootstrap.cc:654] T 7c1e783f0fa44ab98db665a30353573e P 59eeadf53cb043659a02cba07a4300ef: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:25.860225 25059 tablet_bootstrap.cc:492] T 7c1e783f0fa44ab98db665a30353573e P 59eeadf53cb043659a02cba07a4300ef: No bootstrap required, opened a new log
I20260928 06:28:25.860281 25059 ts_tablet_manager.cc:1397] T 7c1e783f0fa44ab98db665a30353573e P 59eeadf53cb043659a02cba07a4300ef: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:28:25.860440 25079 ts_tablet_manager.cc:1918] T b91a57c9f0e344f98d893fcbda78fa32 P c439d4c662d84a308470ceba08705718: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:28:25.860561 25079 log.cc:1196] T b91a57c9f0e344f98d893fcbda78fa32 P c439d4c662d84a308470ceba08705718: Deleting WAL directory at /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/ts-1/wal/wals/b91a57c9f0e344f98d893fcbda78fa32
I20260928 06:28:25.860731 25059 raft_consensus.cc:348] T 7c1e783f0fa44ab98db665a30353573e P 59eeadf53cb043659a02cba07a4300ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59eeadf53cb043659a02cba07a4300ef" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 43685 } } peers { permanent_uuid: "c439d4c662d84a308470ceba08705718" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 39959 } } peers { permanent_uuid: "d2bbaa906b854eaeb416b9b294bda97f" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40979 } }
I20260928 06:28:25.860819 25059 raft_consensus.cc:374] T 7c1e783f0fa44ab98db665a30353573e P 59eeadf53cb043659a02cba07a4300ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:25.860848 25059 raft_consensus.cc:729] T 7c1e783f0fa44ab98db665a30353573e P 59eeadf53cb043659a02cba07a4300ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59eeadf53cb043659a02cba07a4300ef, State: Initialized, Role: FOLLOWER
I20260928 06:28:25.860900 25059 consensus_queue.cc:260] T 7c1e783f0fa44ab98db665a30353573e P 59eeadf53cb043659a02cba07a4300ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59eeadf53cb043659a02cba07a4300ef" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 43685 } } peers { permanent_uuid: "c439d4c662d84a308470ceba08705718" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 39959 } } peers { permanent_uuid: "d2bbaa906b854eaeb416b9b294bda97f" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40979 } }
I20260928 06:28:25.861009 25059 ts_tablet_manager.cc:1428] T 7c1e783f0fa44ab98db665a30353573e P 59eeadf53cb043659a02cba07a4300ef: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 06:28:25.861124 25053 tablet_bootstrap.cc:492] T 7c1e783f0fa44ab98db665a30353573e P d2bbaa906b854eaeb416b9b294bda97f: Bootstrap starting.
I20260928 06:28:25.861552 25053 tablet_bootstrap.cc:654] T 7c1e783f0fa44ab98db665a30353573e P d2bbaa906b854eaeb416b9b294bda97f: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:25.862334 25053 tablet_bootstrap.cc:492] T 7c1e783f0fa44ab98db665a30353573e P d2bbaa906b854eaeb416b9b294bda97f: No bootstrap required, opened a new log
I20260928 06:28:25.862386 25053 ts_tablet_manager.cc:1397] T 7c1e783f0fa44ab98db665a30353573e P d2bbaa906b854eaeb416b9b294bda97f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260928 06:28:25.862556 25053 raft_consensus.cc:348] T 7c1e783f0fa44ab98db665a30353573e P d2bbaa906b854eaeb416b9b294bda97f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59eeadf53cb043659a02cba07a4300ef" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 43685 } } peers { permanent_uuid: "c439d4c662d84a308470ceba08705718" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 39959 } } peers { permanent_uuid: "d2bbaa906b854eaeb416b9b294bda97f" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40979 } }
I20260928 06:28:25.862628 25053 raft_consensus.cc:374] T 7c1e783f0fa44ab98db665a30353573e P d2bbaa906b854eaeb416b9b294bda97f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:25.862655 25053 raft_consensus.cc:729] T 7c1e783f0fa44ab98db665a30353573e P d2bbaa906b854eaeb416b9b294bda97f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2bbaa906b854eaeb416b9b294bda97f, State: Initialized, Role: FOLLOWER
I20260928 06:28:25.862697 25053 consensus_queue.cc:260] T 7c1e783f0fa44ab98db665a30353573e P d2bbaa906b854eaeb416b9b294bda97f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59eeadf53cb043659a02cba07a4300ef" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 43685 } } peers { permanent_uuid: "c439d4c662d84a308470ceba08705718" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 39959 } } peers { permanent_uuid: "d2bbaa906b854eaeb416b9b294bda97f" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40979 } }
I20260928 06:28:25.862787 25053 ts_tablet_manager.cc:1428] T 7c1e783f0fa44ab98db665a30353573e P d2bbaa906b854eaeb416b9b294bda97f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:25.862964 25079 ts_tablet_manager.cc:1939] T b91a57c9f0e344f98d893fcbda78fa32 P c439d4c662d84a308470ceba08705718: Deleting consensus metadata
I20260928 06:28:25.863394 24531 catalog_manager.cc:4841] TS c439d4c662d84a308470ceba08705718 (127.22.225.130:39959): tablet b91a57c9f0e344f98d893fcbda78fa32 (table test-table [id=0db79e07aeba45edaf64b7b0367117c4]) successfully deleted
I20260928 06:28:25.863682 25077 ts_tablet_manager.cc:1905] T b91a57c9f0e344f98d893fcbda78fa32 P d2bbaa906b854eaeb416b9b294bda97f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:28:25.864717 25078 tablet_replica.cc:330] T b91a57c9f0e344f98d893fcbda78fa32 P 59eeadf53cb043659a02cba07a4300ef: stopping tablet replica
I20260928 06:28:25.864876 25078 raft_consensus.cc:2229] T b91a57c9f0e344f98d893fcbda78fa32 P 59eeadf53cb043659a02cba07a4300ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:28:25.864928 25077 ts_tablet_manager.cc:1918] T b91a57c9f0e344f98d893fcbda78fa32 P d2bbaa906b854eaeb416b9b294bda97f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:28:25.864993 25077 log.cc:1196] T b91a57c9f0e344f98d893fcbda78fa32 P d2bbaa906b854eaeb416b9b294bda97f: Deleting WAL directory at /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/ts-2/wal/wals/b91a57c9f0e344f98d893fcbda78fa32
I20260928 06:28:25.865162 25078 raft_consensus.cc:2258] T b91a57c9f0e344f98d893fcbda78fa32 P 59eeadf53cb043659a02cba07a4300ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:28:25.865247 25077 ts_tablet_manager.cc:1939] T b91a57c9f0e344f98d893fcbda78fa32 P d2bbaa906b854eaeb416b9b294bda97f: Deleting consensus metadata
I20260928 06:28:25.865818 25078 ts_tablet_manager.cc:1905] T b91a57c9f0e344f98d893fcbda78fa32 P 59eeadf53cb043659a02cba07a4300ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:28:25.865928 24534 catalog_manager.cc:4841] TS d2bbaa906b854eaeb416b9b294bda97f (127.22.225.131:40979): tablet b91a57c9f0e344f98d893fcbda78fa32 (table test-table [id=0db79e07aeba45edaf64b7b0367117c4]) successfully deleted
I20260928 06:28:25.867216 25078 ts_tablet_manager.cc:1918] T b91a57c9f0e344f98d893fcbda78fa32 P 59eeadf53cb043659a02cba07a4300ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:28:25.867339 25078 log.cc:1196] T b91a57c9f0e344f98d893fcbda78fa32 P 59eeadf53cb043659a02cba07a4300ef: Deleting WAL directory at /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/ts-0/wal/wals/b91a57c9f0e344f98d893fcbda78fa32
I20260928 06:28:25.867455 25054 tablet_bootstrap.cc:492] T 7c1e783f0fa44ab98db665a30353573e P c439d4c662d84a308470ceba08705718: No bootstrap required, opened a new log
I20260928 06:28:25.867528 25054 ts_tablet_manager.cc:1397] T 7c1e783f0fa44ab98db665a30353573e P c439d4c662d84a308470ceba08705718: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.001s
I20260928 06:28:25.867645 25078 ts_tablet_manager.cc:1939] T b91a57c9f0e344f98d893fcbda78fa32 P 59eeadf53cb043659a02cba07a4300ef: Deleting consensus metadata
I20260928 06:28:25.867666 25054 raft_consensus.cc:348] T 7c1e783f0fa44ab98db665a30353573e P c439d4c662d84a308470ceba08705718 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59eeadf53cb043659a02cba07a4300ef" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 43685 } } peers { permanent_uuid: "c439d4c662d84a308470ceba08705718" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 39959 } } peers { permanent_uuid: "d2bbaa906b854eaeb416b9b294bda97f" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40979 } }
I20260928 06:28:25.867750 25054 raft_consensus.cc:374] T 7c1e783f0fa44ab98db665a30353573e P c439d4c662d84a308470ceba08705718 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:25.867779 25054 raft_consensus.cc:729] T 7c1e783f0fa44ab98db665a30353573e P c439d4c662d84a308470ceba08705718 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c439d4c662d84a308470ceba08705718, State: Initialized, Role: FOLLOWER
I20260928 06:28:25.867828 25054 consensus_queue.cc:260] T 7c1e783f0fa44ab98db665a30353573e P c439d4c662d84a308470ceba08705718 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59eeadf53cb043659a02cba07a4300ef" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 43685 } } peers { permanent_uuid: "c439d4c662d84a308470ceba08705718" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 39959 } } peers { permanent_uuid: "d2bbaa906b854eaeb416b9b294bda97f" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40979 } }
I20260928 06:28:25.867942 25054 ts_tablet_manager.cc:1428] T 7c1e783f0fa44ab98db665a30353573e P c439d4c662d84a308470ceba08705718: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:25.868121 24533 catalog_manager.cc:4841] TS 59eeadf53cb043659a02cba07a4300ef (127.22.225.129:43685): tablet b91a57c9f0e344f98d893fcbda78fa32 (table test-table [id=0db79e07aeba45edaf64b7b0367117c4]) successfully deleted
I20260928 06:28:25.944470 25061 raft_consensus.cc:482] T 7c1e783f0fa44ab98db665a30353573e P 59eeadf53cb043659a02cba07a4300ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:28:25.944604 25061 raft_consensus.cc:504] T 7c1e783f0fa44ab98db665a30353573e P 59eeadf53cb043659a02cba07a4300ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59eeadf53cb043659a02cba07a4300ef" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 43685 } } peers { permanent_uuid: "c439d4c662d84a308470ceba08705718" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 39959 } } peers { permanent_uuid: "d2bbaa906b854eaeb416b9b294bda97f" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40979 } }
I20260928 06:28:25.944984 25061 leader_election.cc:288] T 7c1e783f0fa44ab98db665a30353573e P 59eeadf53cb043659a02cba07a4300ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c439d4c662d84a308470ceba08705718 (127.22.225.130:39959), d2bbaa906b854eaeb416b9b294bda97f (127.22.225.131:40979)
I20260928 06:28:25.949030 25052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:25.945366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:58649 (local address 127.22.225.130:39959)
0928 06:28:25.945409 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:25.945411 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:25.945422 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:28:25.945490 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:25.945491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:25.945499 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:25.945558 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:28:25.945750 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:25.946555 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:25.948055 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:25.948632 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:25.948679 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:25.948688 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:25.948916 (+ 228us) server_negotiation.cc:299] Negotiation successful
0928 06:28:25.948956 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 06:28:25.949211 25050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:25.945426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:54877 (local address 127.22.225.131:40979)
0928 06:28:25.945470 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:25.945471 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:25.945645 (+ 174us) server_negotiation.cc:407] Connection header received
0928 06:28:25.945749 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:25.945750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:25.945760 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:25.945789 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:28:25.946063 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:25.946753 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:25.948222 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:25.948783 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:25.948835 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:25.948843 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:25.949106 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 06:28:25.949142 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:28:25.949566 25084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:25.945252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.131:40979 (local address 127.22.225.129:54877)
0928 06:28:25.945577 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:25.945587 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:25.945672 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:25.945902 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:25.945902 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:25.945904 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:28:25.945974 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:25.945975 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:25.947035 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:25.947036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:25.948116 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:25.948119 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:25.948935 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:25.948937 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:25.949017 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:25.949021 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:28:25.949441 (+ 420us) client_negotiation.cc:240] Negotiation successful
0928 06:28:25.949486 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":73,"threads_started":1}
I20260928 06:28:25.949584 25083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:25.945228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.130:39959 (local address 127.22.225.129:58649)
0928 06:28:25.945340 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:25.945354 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:25.945415 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:25.945604 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:25.945605 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:25.945607 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:28:25.945682 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:25.945683 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:25.946667 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:25.946668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:25.947948 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:25.947951 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:25.948743 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:25.948744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:25.948839 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:25.948842 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:28:25.949417 (+ 575us) client_negotiation.cc:240] Negotiation successful
0928 06:28:25.949469 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":71,"threads_started":1}
I20260928 06:28:25.949818 24984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c1e783f0fa44ab98db665a30353573e" candidate_uuid: "59eeadf53cb043659a02cba07a4300ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2bbaa906b854eaeb416b9b294bda97f" is_pre_election: true
I20260928 06:28:25.949916 24984 raft_consensus.cc:2454] T 7c1e783f0fa44ab98db665a30353573e P d2bbaa906b854eaeb416b9b294bda97f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59eeadf53cb043659a02cba07a4300ef in term 0.
I20260928 06:28:25.950026 24844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c1e783f0fa44ab98db665a30353573e" candidate_uuid: "59eeadf53cb043659a02cba07a4300ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c439d4c662d84a308470ceba08705718" is_pre_election: true
I20260928 06:28:25.950109 24844 raft_consensus.cc:2454] T 7c1e783f0fa44ab98db665a30353573e P c439d4c662d84a308470ceba08705718 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59eeadf53cb043659a02cba07a4300ef in term 0.
I20260928 06:28:25.950227 24638 leader_election.cc:302] T 7c1e783f0fa44ab98db665a30353573e P 59eeadf53cb043659a02cba07a4300ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59eeadf53cb043659a02cba07a4300ef, d2bbaa906b854eaeb416b9b294bda97f; no voters:
I20260928 06:28:25.950348 25061 raft_consensus.cc:2781] T 7c1e783f0fa44ab98db665a30353573e P 59eeadf53cb043659a02cba07a4300ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:28:25.950380 25061 raft_consensus.cc:482] T 7c1e783f0fa44ab98db665a30353573e P 59eeadf53cb043659a02cba07a4300ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:28:25.950402 25061 raft_consensus.cc:3037] T 7c1e783f0fa44ab98db665a30353573e P 59eeadf53cb043659a02cba07a4300ef [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:25.951278 25061 raft_consensus.cc:504] T 7c1e783f0fa44ab98db665a30353573e P 59eeadf53cb043659a02cba07a4300ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59eeadf53cb043659a02cba07a4300ef" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 43685 } } peers { permanent_uuid: "c439d4c662d84a308470ceba08705718" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 39959 } } peers { permanent_uuid: "d2bbaa906b854eaeb416b9b294bda97f" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40979 } }
I20260928 06:28:25.951416 25061 leader_election.cc:288] T 7c1e783f0fa44ab98db665a30353573e P 59eeadf53cb043659a02cba07a4300ef [CANDIDATE]: Term 1 election: Requested vote from peers c439d4c662d84a308470ceba08705718 (127.22.225.130:39959), d2bbaa906b854eaeb416b9b294bda97f (127.22.225.131:40979)
I20260928 06:28:25.951726 24844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c1e783f0fa44ab98db665a30353573e" candidate_uuid: "59eeadf53cb043659a02cba07a4300ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c439d4c662d84a308470ceba08705718"
I20260928 06:28:25.951814 24844 raft_consensus.cc:3037] T 7c1e783f0fa44ab98db665a30353573e P c439d4c662d84a308470ceba08705718 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:25.951805 24984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c1e783f0fa44ab98db665a30353573e" candidate_uuid: "59eeadf53cb043659a02cba07a4300ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2bbaa906b854eaeb416b9b294bda97f"
I20260928 06:28:25.951865 24984 raft_consensus.cc:3037] T 7c1e783f0fa44ab98db665a30353573e P d2bbaa906b854eaeb416b9b294bda97f [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:25.952633 24984 raft_consensus.cc:2454] T 7c1e783f0fa44ab98db665a30353573e P d2bbaa906b854eaeb416b9b294bda97f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59eeadf53cb043659a02cba07a4300ef in term 1.
I20260928 06:28:25.952633 24844 raft_consensus.cc:2454] T 7c1e783f0fa44ab98db665a30353573e P c439d4c662d84a308470ceba08705718 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59eeadf53cb043659a02cba07a4300ef in term 1.
I20260928 06:28:25.952796 24638 leader_election.cc:302] T 7c1e783f0fa44ab98db665a30353573e P 59eeadf53cb043659a02cba07a4300ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59eeadf53cb043659a02cba07a4300ef, d2bbaa906b854eaeb416b9b294bda97f; no voters:
I20260928 06:28:25.952898 25061 raft_consensus.cc:2781] T 7c1e783f0fa44ab98db665a30353573e P 59eeadf53cb043659a02cba07a4300ef [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:28:25.953039 25061 raft_consensus.cc:686] T 7c1e783f0fa44ab98db665a30353573e P 59eeadf53cb043659a02cba07a4300ef [term 1 LEADER]: Becoming Leader. State: Replica: 59eeadf53cb043659a02cba07a4300ef, State: Running, Role: LEADER
I20260928 06:28:25.953104 25061 consensus_queue.cc:237] T 7c1e783f0fa44ab98db665a30353573e P 59eeadf53cb043659a02cba07a4300ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59eeadf53cb043659a02cba07a4300ef" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 43685 } } peers { permanent_uuid: "c439d4c662d84a308470ceba08705718" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 39959 } } peers { permanent_uuid: "d2bbaa906b854eaeb416b9b294bda97f" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40979 } }
I20260928 06:28:25.953768 24547 catalog_manager.cc:5495] T 7c1e783f0fa44ab98db665a30353573e P 59eeadf53cb043659a02cba07a4300ef reported cstate change: term changed from 0 to 1, leader changed from <none> to 59eeadf53cb043659a02cba07a4300ef (127.22.225.129). New cstate: current_term: 1 leader_uuid: "59eeadf53cb043659a02cba07a4300ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59eeadf53cb043659a02cba07a4300ef" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 43685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c439d4c662d84a308470ceba08705718" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 39959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2bbaa906b854eaeb416b9b294bda97f" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40979 } health_report { overall_health: UNKNOWN } } }
Sep 28 06:28:26 dist-test-slave-d59m krb5kdc[24502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576904, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.190@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:28:26.021520 24596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:26.009523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37830 (local address 127.22.225.190:46621)
0928 06:28:26.009593 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:26.009596 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:26.009715 (+ 119us) server_negotiation.cc:407] Connection header received
0928 06:28:26.009800 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:26.009801 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:26.009815 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:26.009886 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:26.010775 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:26.011716 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:26.013239 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:26.013440 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:26.013526 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:26.018830 (+ 5304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:26.018832 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:26.018832 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:26.018840 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:26.020363 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:26.020801 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:26.020802 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:26.020802 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:26.020850 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:26.021029 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:26.021030 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:26.021030 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:26.021142 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:26.021186 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:26.021336 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 06:28:26.021425 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 06:28:26.021776 25092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:26.009454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:46621 (local address 127.0.0.1:37830)
0928 06:28:26.009629 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:26.009648 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:26.009736 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:26.009956 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:26.009957 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:26.010566 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:26.010638 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:26.010643 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:26.011819 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:26.011820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:26.013152 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:26.013157 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:26.013526 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:26.013527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:26.013577 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:26.016902 (+ 3325us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:26.016906 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:26.018718 (+ 1812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:26.020455 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:26.020463 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:26.020465 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:26.020725 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:26.020926 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:26.020927 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:26.020927 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:26.020977 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:26.021209 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:26.021214 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:26.021250 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 06:28:26.021607 (+ 357us) client_negotiation.cc:240] Negotiation successful
0928 06:28:26.021698 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":103,"threads_started":1}
I20260928 06:28:26.024116 24547 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.22.225.190@KRBTEST.COM'} at 127.0.0.1:37830:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 06:28:26.024338 24547 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:28:26.025777 24824 tablet_service.cc:1467] Processing CreateTablet for tablet 0abccd76bde34c7a960e8df4342ff1f8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9174eed70b604e7895bcb96a42abf16d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 06:28:26.025797 24684 tablet_service.cc:1467] Processing CreateTablet for tablet 0abccd76bde34c7a960e8df4342ff1f8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9174eed70b604e7895bcb96a42abf16d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 06:28:26.025797 24964 tablet_service.cc:1467] Processing CreateTablet for tablet 0abccd76bde34c7a960e8df4342ff1f8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9174eed70b604e7895bcb96a42abf16d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 06:28:26.025956 24824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0abccd76bde34c7a960e8df4342ff1f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:26.025964 24964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0abccd76bde34c7a960e8df4342ff1f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:26.025964 24684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0abccd76bde34c7a960e8df4342ff1f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:26.027403 25054 tablet_bootstrap.cc:492] T 0abccd76bde34c7a960e8df4342ff1f8 P c439d4c662d84a308470ceba08705718: Bootstrap starting.
I20260928 06:28:26.027467 25053 tablet_bootstrap.cc:492] T 0abccd76bde34c7a960e8df4342ff1f8 P d2bbaa906b854eaeb416b9b294bda97f: Bootstrap starting.
I20260928 06:28:26.027791 25053 tablet_bootstrap.cc:654] T 0abccd76bde34c7a960e8df4342ff1f8 P d2bbaa906b854eaeb416b9b294bda97f: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:26.027833 25054 tablet_bootstrap.cc:654] T 0abccd76bde34c7a960e8df4342ff1f8 P c439d4c662d84a308470ceba08705718: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:26.028079 25059 tablet_bootstrap.cc:492] T 0abccd76bde34c7a960e8df4342ff1f8 P 59eeadf53cb043659a02cba07a4300ef: Bootstrap starting.
I20260928 06:28:26.028410 25054 tablet_bootstrap.cc:492] T 0abccd76bde34c7a960e8df4342ff1f8 P c439d4c662d84a308470ceba08705718: No bootstrap required, opened a new log
I20260928 06:28:26.028446 25054 ts_tablet_manager.cc:1397] T 0abccd76bde34c7a960e8df4342ff1f8 P c439d4c662d84a308470ceba08705718: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 06:28:26.028471 25059 tablet_bootstrap.cc:654] T 0abccd76bde34c7a960e8df4342ff1f8 P 59eeadf53cb043659a02cba07a4300ef: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:26.028549 25054 raft_consensus.cc:348] T 0abccd76bde34c7a960e8df4342ff1f8 P c439d4c662d84a308470ceba08705718 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59eeadf53cb043659a02cba07a4300ef" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 43685 } } peers { permanent_uuid: "c439d4c662d84a308470ceba08705718" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 39959 } } peers { permanent_uuid: "d2bbaa906b854eaeb416b9b294bda97f" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40979 } }
I20260928 06:28:26.028676 25054 raft_consensus.cc:374] T 0abccd76bde34c7a960e8df4342ff1f8 P c439d4c662d84a308470ceba08705718 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:26.028695 25054 raft_consensus.cc:729] T 0abccd76bde34c7a960e8df4342ff1f8 P c439d4c662d84a308470ceba08705718 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c439d4c662d84a308470ceba08705718, State: Initialized, Role: FOLLOWER
I20260928 06:28:26.028756 25054 consensus_queue.cc:260] T 0abccd76bde34c7a960e8df4342ff1f8 P c439d4c662d84a308470ceba08705718 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59eeadf53cb043659a02cba07a4300ef" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 43685 } } peers { permanent_uuid: "c439d4c662d84a308470ceba08705718" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 39959 } } peers { permanent_uuid: "d2bbaa906b854eaeb416b9b294bda97f" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40979 } }
I20260928 06:28:26.028857 25054 ts_tablet_manager.cc:1428] T 0abccd76bde34c7a960e8df4342ff1f8 P c439d4c662d84a308470ceba08705718: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 06:28:26.028981 25058 tablet_replica.cc:439] T 0abccd76bde34c7a960e8df4342ff1f8 P c439d4c662d84a308470ceba08705718: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59eeadf53cb043659a02cba07a4300ef" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 43685 } } peers { permanent_uuid: "c439d4c662d84a308470ceba08705718" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 39959 } } peers { permanent_uuid: "d2bbaa906b854eaeb416b9b294bda97f" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40979 } } }
I20260928 06:28:26.029068 25058 tablet_replica.cc:442] T 0abccd76bde34c7a960e8df4342ff1f8 P c439d4c662d84a308470ceba08705718: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:28:26.029137 25053 tablet_bootstrap.cc:492] T 0abccd76bde34c7a960e8df4342ff1f8 P d2bbaa906b854eaeb416b9b294bda97f: No bootstrap required, opened a new log
I20260928 06:28:26.029187 25053 ts_tablet_manager.cc:1397] T 0abccd76bde34c7a960e8df4342ff1f8 P d2bbaa906b854eaeb416b9b294bda97f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20260928 06:28:26.029201 25059 tablet_bootstrap.cc:492] T 0abccd76bde34c7a960e8df4342ff1f8 P 59eeadf53cb043659a02cba07a4300ef: No bootstrap required, opened a new log
I20260928 06:28:26.029258 25059 ts_tablet_manager.cc:1397] T 0abccd76bde34c7a960e8df4342ff1f8 P 59eeadf53cb043659a02cba07a4300ef: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 06:28:26.029301 25053 raft_consensus.cc:348] T 0abccd76bde34c7a960e8df4342ff1f8 P d2bbaa906b854eaeb416b9b294bda97f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59eeadf53cb043659a02cba07a4300ef" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 43685 } } peers { permanent_uuid: "c439d4c662d84a308470ceba08705718" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 39959 } } peers { permanent_uuid: "d2bbaa906b854eaeb416b9b294bda97f" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40979 } }
I20260928 06:28:26.029368 25053 raft_consensus.cc:374] T 0abccd76bde34c7a960e8df4342ff1f8 P d2bbaa906b854eaeb416b9b294bda97f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:26.029386 25053 raft_consensus.cc:729] T 0abccd76bde34c7a960e8df4342ff1f8 P d2bbaa906b854eaeb416b9b294bda97f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2bbaa906b854eaeb416b9b294bda97f, State: Initialized, Role: FOLLOWER
I20260928 06:28:26.029424 25053 consensus_queue.cc:260] T 0abccd76bde34c7a960e8df4342ff1f8 P d2bbaa906b854eaeb416b9b294bda97f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59eeadf53cb043659a02cba07a4300ef" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 43685 } } peers { permanent_uuid: "c439d4c662d84a308470ceba08705718" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 39959 } } peers { permanent_uuid: "d2bbaa906b854eaeb416b9b294bda97f" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40979 } }
I20260928 06:28:26.029415 25059 raft_consensus.cc:348] T 0abccd76bde34c7a960e8df4342ff1f8 P 59eeadf53cb043659a02cba07a4300ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59eeadf53cb043659a02cba07a4300ef" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 43685 } } peers { permanent_uuid: "c439d4c662d84a308470ceba08705718" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 39959 } } peers { permanent_uuid: "d2bbaa906b854eaeb416b9b294bda97f" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40979 } }
I20260928 06:28:26.029476 25059 raft_consensus.cc:374] T 0abccd76bde34c7a960e8df4342ff1f8 P 59eeadf53cb043659a02cba07a4300ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:26.029495 25059 raft_consensus.cc:729] T 0abccd76bde34c7a960e8df4342ff1f8 P 59eeadf53cb043659a02cba07a4300ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59eeadf53cb043659a02cba07a4300ef, State: Initialized, Role: FOLLOWER
I20260928 06:28:26.029510 25053 ts_tablet_manager.cc:1428] T 0abccd76bde34c7a960e8df4342ff1f8 P d2bbaa906b854eaeb416b9b294bda97f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:26.029533 25059 consensus_queue.cc:260] T 0abccd76bde34c7a960e8df4342ff1f8 P 59eeadf53cb043659a02cba07a4300ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59eeadf53cb043659a02cba07a4300ef" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 43685 } } peers { permanent_uuid: "c439d4c662d84a308470ceba08705718" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 39959 } } peers { permanent_uuid: "d2bbaa906b854eaeb416b9b294bda97f" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40979 } }
I20260928 06:28:26.029623 25059 ts_tablet_manager.cc:1428] T 0abccd76bde34c7a960e8df4342ff1f8 P 59eeadf53cb043659a02cba07a4300ef: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:26.029621 25065 tablet_replica.cc:439] T 0abccd76bde34c7a960e8df4342ff1f8 P d2bbaa906b854eaeb416b9b294bda97f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59eeadf53cb043659a02cba07a4300ef" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 43685 } } peers { permanent_uuid: "c439d4c662d84a308470ceba08705718" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 39959 } } peers { permanent_uuid: "d2bbaa906b854eaeb416b9b294bda97f" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40979 } } }
I20260928 06:28:26.029698 25065 tablet_replica.cc:442] T 0abccd76bde34c7a960e8df4342ff1f8 P d2bbaa906b854eaeb416b9b294bda97f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:28:26.029892 25061 tablet_replica.cc:439] T 0abccd76bde34c7a960e8df4342ff1f8 P 59eeadf53cb043659a02cba07a4300ef: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59eeadf53cb043659a02cba07a4300ef" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 43685 } } peers { permanent_uuid: "c439d4c662d84a308470ceba08705718" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 39959 } } peers { permanent_uuid: "d2bbaa906b854eaeb416b9b294bda97f" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40979 } } }
I20260928 06:28:26.029982 25061 tablet_replica.cc:442] T 0abccd76bde34c7a960e8df4342ff1f8 P 59eeadf53cb043659a02cba07a4300ef: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:28:26.133920 25061 raft_consensus.cc:482] T 0abccd76bde34c7a960e8df4342ff1f8 P 59eeadf53cb043659a02cba07a4300ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:28:26.134042 25061 raft_consensus.cc:504] T 0abccd76bde34c7a960e8df4342ff1f8 P 59eeadf53cb043659a02cba07a4300ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59eeadf53cb043659a02cba07a4300ef" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 43685 } } peers { permanent_uuid: "c439d4c662d84a308470ceba08705718" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 39959 } } peers { permanent_uuid: "d2bbaa906b854eaeb416b9b294bda97f" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40979 } }
I20260928 06:28:26.134208 25061 leader_election.cc:288] T 0abccd76bde34c7a960e8df4342ff1f8 P 59eeadf53cb043659a02cba07a4300ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c439d4c662d84a308470ceba08705718 (127.22.225.130:39959), d2bbaa906b854eaeb416b9b294bda97f (127.22.225.131:40979)
I20260928 06:28:26.134466 24844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0abccd76bde34c7a960e8df4342ff1f8" candidate_uuid: "59eeadf53cb043659a02cba07a4300ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c439d4c662d84a308470ceba08705718" is_pre_election: true
I20260928 06:28:26.134471 24984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0abccd76bde34c7a960e8df4342ff1f8" candidate_uuid: "59eeadf53cb043659a02cba07a4300ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2bbaa906b854eaeb416b9b294bda97f" is_pre_election: true
I20260928 06:28:26.134580 24844 raft_consensus.cc:2454] T 0abccd76bde34c7a960e8df4342ff1f8 P c439d4c662d84a308470ceba08705718 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59eeadf53cb043659a02cba07a4300ef in term 0.
I20260928 06:28:26.134590 24984 raft_consensus.cc:2454] T 0abccd76bde34c7a960e8df4342ff1f8 P d2bbaa906b854eaeb416b9b294bda97f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59eeadf53cb043659a02cba07a4300ef in term 0.
I20260928 06:28:26.134764 24638 leader_election.cc:302] T 0abccd76bde34c7a960e8df4342ff1f8 P 59eeadf53cb043659a02cba07a4300ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59eeadf53cb043659a02cba07a4300ef, d2bbaa906b854eaeb416b9b294bda97f; no voters:
I20260928 06:28:26.134881 25061 raft_consensus.cc:2781] T 0abccd76bde34c7a960e8df4342ff1f8 P 59eeadf53cb043659a02cba07a4300ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:28:26.134925 25061 raft_consensus.cc:482] T 0abccd76bde34c7a960e8df4342ff1f8 P 59eeadf53cb043659a02cba07a4300ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:28:26.134974 25061 raft_consensus.cc:3037] T 0abccd76bde34c7a960e8df4342ff1f8 P 59eeadf53cb043659a02cba07a4300ef [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:26.135699 25061 raft_consensus.cc:504] T 0abccd76bde34c7a960e8df4342ff1f8 P 59eeadf53cb043659a02cba07a4300ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59eeadf53cb043659a02cba07a4300ef" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 43685 } } peers { permanent_uuid: "c439d4c662d84a308470ceba08705718" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 39959 } } peers { permanent_uuid: "d2bbaa906b854eaeb416b9b294bda97f" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40979 } }
I20260928 06:28:26.135840 25061 leader_election.cc:288] T 0abccd76bde34c7a960e8df4342ff1f8 P 59eeadf53cb043659a02cba07a4300ef [CANDIDATE]: Term 1 election: Requested vote from peers c439d4c662d84a308470ceba08705718 (127.22.225.130:39959), d2bbaa906b854eaeb416b9b294bda97f (127.22.225.131:40979)
I20260928 06:28:26.136024 24844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0abccd76bde34c7a960e8df4342ff1f8" candidate_uuid: "59eeadf53cb043659a02cba07a4300ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c439d4c662d84a308470ceba08705718"
I20260928 06:28:26.136031 24984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0abccd76bde34c7a960e8df4342ff1f8" candidate_uuid: "59eeadf53cb043659a02cba07a4300ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2bbaa906b854eaeb416b9b294bda97f"
I20260928 06:28:26.136126 24984 raft_consensus.cc:3037] T 0abccd76bde34c7a960e8df4342ff1f8 P d2bbaa906b854eaeb416b9b294bda97f [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:26.136128 24844 raft_consensus.cc:3037] T 0abccd76bde34c7a960e8df4342ff1f8 P c439d4c662d84a308470ceba08705718 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:26.136847 24844 raft_consensus.cc:2454] T 0abccd76bde34c7a960e8df4342ff1f8 P c439d4c662d84a308470ceba08705718 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59eeadf53cb043659a02cba07a4300ef in term 1.
I20260928 06:28:26.136958 24984 raft_consensus.cc:2454] T 0abccd76bde34c7a960e8df4342ff1f8 P d2bbaa906b854eaeb416b9b294bda97f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59eeadf53cb043659a02cba07a4300ef in term 1.
I20260928 06:28:26.137029 24635 leader_election.cc:302] T 0abccd76bde34c7a960e8df4342ff1f8 P 59eeadf53cb043659a02cba07a4300ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59eeadf53cb043659a02cba07a4300ef, c439d4c662d84a308470ceba08705718; no voters:
I20260928 06:28:26.137123 25061 raft_consensus.cc:2781] T 0abccd76bde34c7a960e8df4342ff1f8 P 59eeadf53cb043659a02cba07a4300ef [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:28:26.137176 25061 raft_consensus.cc:686] T 0abccd76bde34c7a960e8df4342ff1f8 P 59eeadf53cb043659a02cba07a4300ef [term 1 LEADER]: Becoming Leader. State: Replica: 59eeadf53cb043659a02cba07a4300ef, State: Running, Role: LEADER
I20260928 06:28:26.137225 25061 consensus_queue.cc:237] T 0abccd76bde34c7a960e8df4342ff1f8 P 59eeadf53cb043659a02cba07a4300ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59eeadf53cb043659a02cba07a4300ef" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 43685 } } peers { permanent_uuid: "c439d4c662d84a308470ceba08705718" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 39959 } } peers { permanent_uuid: "d2bbaa906b854eaeb416b9b294bda97f" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40979 } }
I20260928 06:28:26.137485 25085 tablet_replica.cc:439] T 0abccd76bde34c7a960e8df4342ff1f8 P 59eeadf53cb043659a02cba07a4300ef: TxnStatusTablet state changed. Reason: New leader 59eeadf53cb043659a02cba07a4300ef. Latest consensus state: current_term: 1 leader_uuid: "59eeadf53cb043659a02cba07a4300ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59eeadf53cb043659a02cba07a4300ef" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 43685 } } peers { permanent_uuid: "c439d4c662d84a308470ceba08705718" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 39959 } } peers { permanent_uuid: "d2bbaa906b854eaeb416b9b294bda97f" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40979 } } }
I20260928 06:28:26.137566 25085 tablet_replica.cc:442] T 0abccd76bde34c7a960e8df4342ff1f8 P 59eeadf53cb043659a02cba07a4300ef: This TxnStatusTablet replica's current role is: LEADER
I20260928 06:28:26.137812 25097 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 06:28:26.137809 24546 catalog_manager.cc:5495] T 0abccd76bde34c7a960e8df4342ff1f8 P 59eeadf53cb043659a02cba07a4300ef reported cstate change: term changed from 0 to 1, leader changed from <none> to 59eeadf53cb043659a02cba07a4300ef (127.22.225.129). New cstate: current_term: 1 leader_uuid: "59eeadf53cb043659a02cba07a4300ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59eeadf53cb043659a02cba07a4300ef" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 43685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c439d4c662d84a308470ceba08705718" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 39959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2bbaa906b854eaeb416b9b294bda97f" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40979 } health_report { overall_health: UNKNOWN } } }
I20260928 06:28:26.137939 25097 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 06:28:26.146991 25092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:26.143261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.129:43685 (local address 127.0.0.1:49224)
0928 06:28:26.143424 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:26.143437 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:26.143484 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:26.143818 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:26.143819 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:26.143821 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:28:26.143871 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:26.143874 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:26.144925 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:26.144926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:26.145850 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:26.145852 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:26.146384 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:26.146385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:26.146443 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:26.146461 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:28:26.146793 (+ 332us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:28:26.146803 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 06:28:26.146848 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 06:28:26.146897 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105}
I20260928 06:28:26.147601 25064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:26.143582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49224 (local address 127.22.225.129:43685)
0928 06:28:26.143645 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:26.143647 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:26.143654 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:28:26.143684 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:26.143685 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:26.143704 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:26.143761 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:28:26.143942 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:26.144830 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:26.145974 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:26.146280 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:26.146344 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:26.146569 (+ 225us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:28:26.146697 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:28:26.146765 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:26.147508 (+ 743us) server_negotiation.cc:299] Negotiation successful
0928 06:28:26.147535 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 06:28:26.148684 24844 raft_consensus.cc:1261] T 0abccd76bde34c7a960e8df4342ff1f8 P c439d4c662d84a308470ceba08705718 [term 1 FOLLOWER]: Refusing update from remote peer 59eeadf53cb043659a02cba07a4300ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:28:26.148749 24984 raft_consensus.cc:1261] T 0abccd76bde34c7a960e8df4342ff1f8 P d2bbaa906b854eaeb416b9b294bda97f [term 1 FOLLOWER]: Refusing update from remote peer 59eeadf53cb043659a02cba07a4300ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:28:26.148921 25085 consensus_queue.cc:1035] T 0abccd76bde34c7a960e8df4342ff1f8 P 59eeadf53cb043659a02cba07a4300ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "c439d4c662d84a308470ceba08705718" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 39959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:28:26.149011 25061 consensus_queue.cc:1035] T 0abccd76bde34c7a960e8df4342ff1f8 P 59eeadf53cb043659a02cba07a4300ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2bbaa906b854eaeb416b9b294bda97f" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:28:26.149621 25065 tablet_replica.cc:439] T 0abccd76bde34c7a960e8df4342ff1f8 P d2bbaa906b854eaeb416b9b294bda97f: TxnStatusTablet state changed. Reason: New leader 59eeadf53cb043659a02cba07a4300ef. Latest consensus state: current_term: 1 leader_uuid: "59eeadf53cb043659a02cba07a4300ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59eeadf53cb043659a02cba07a4300ef" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 43685 } } peers { permanent_uuid: "c439d4c662d84a308470ceba08705718" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 39959 } } peers { permanent_uuid: "d2bbaa906b854eaeb416b9b294bda97f" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40979 } } }
I20260928 06:28:26.149717 25065 tablet_replica.cc:442] T 0abccd76bde34c7a960e8df4342ff1f8 P d2bbaa906b854eaeb416b9b294bda97f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:28:26.149696 25058 tablet_replica.cc:439] T 0abccd76bde34c7a960e8df4342ff1f8 P c439d4c662d84a308470ceba08705718: TxnStatusTablet state changed. Reason: New leader 59eeadf53cb043659a02cba07a4300ef. Latest consensus state: current_term: 1 leader_uuid: "59eeadf53cb043659a02cba07a4300ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59eeadf53cb043659a02cba07a4300ef" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 43685 } } peers { permanent_uuid: "c439d4c662d84a308470ceba08705718" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 39959 } } peers { permanent_uuid: "d2bbaa906b854eaeb416b9b294bda97f" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40979 } } }
I20260928 06:28:26.149785 25058 tablet_replica.cc:442] T 0abccd76bde34c7a960e8df4342ff1f8 P c439d4c662d84a308470ceba08705718: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:28:26.150513 25085 tablet_replica.cc:439] T 0abccd76bde34c7a960e8df4342ff1f8 P 59eeadf53cb043659a02cba07a4300ef: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "59eeadf53cb043659a02cba07a4300ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59eeadf53cb043659a02cba07a4300ef" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 43685 } } peers { permanent_uuid: "c439d4c662d84a308470ceba08705718" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 39959 } } peers { permanent_uuid: "d2bbaa906b854eaeb416b9b294bda97f" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40979 } } }
I20260928 06:28:26.150606 25085 tablet_replica.cc:442] T 0abccd76bde34c7a960e8df4342ff1f8 P 59eeadf53cb043659a02cba07a4300ef: This TxnStatusTablet replica's current role is: LEADER
I20260928 06:28:26.150929 25065 tablet_replica.cc:439] T 0abccd76bde34c7a960e8df4342ff1f8 P d2bbaa906b854eaeb416b9b294bda97f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "59eeadf53cb043659a02cba07a4300ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59eeadf53cb043659a02cba07a4300ef" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 43685 } } peers { permanent_uuid: "c439d4c662d84a308470ceba08705718" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 39959 } } peers { permanent_uuid: "d2bbaa906b854eaeb416b9b294bda97f" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40979 } } }
I20260928 06:28:26.150996 25065 tablet_replica.cc:442] T 0abccd76bde34c7a960e8df4342ff1f8 P d2bbaa906b854eaeb416b9b294bda97f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:28:26.150983 25061 tablet_replica.cc:439] T 0abccd76bde34c7a960e8df4342ff1f8 P 59eeadf53cb043659a02cba07a4300ef: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "59eeadf53cb043659a02cba07a4300ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59eeadf53cb043659a02cba07a4300ef" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 43685 } } peers { permanent_uuid: "c439d4c662d84a308470ceba08705718" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 39959 } } peers { permanent_uuid: "d2bbaa906b854eaeb416b9b294bda97f" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40979 } } }
I20260928 06:28:26.151054 25061 tablet_replica.cc:442] T 0abccd76bde34c7a960e8df4342ff1f8 P 59eeadf53cb043659a02cba07a4300ef: This TxnStatusTablet replica's current role is: LEADER
I20260928 06:28:26.152127 25058 tablet_replica.cc:439] T 0abccd76bde34c7a960e8df4342ff1f8 P c439d4c662d84a308470ceba08705718: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "59eeadf53cb043659a02cba07a4300ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59eeadf53cb043659a02cba07a4300ef" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 43685 } } peers { permanent_uuid: "c439d4c662d84a308470ceba08705718" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 39959 } } peers { permanent_uuid: "d2bbaa906b854eaeb416b9b294bda97f" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40979 } } }
I20260928 06:28:26.152216 25058 tablet_replica.cc:442] T 0abccd76bde34c7a960e8df4342ff1f8 P c439d4c662d84a308470ceba08705718: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:28:26.156807 25064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:26.153719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49230 (local address 127.22.225.129:43685)
0928 06:28:26.153758 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:26.153760 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:26.153767 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:28:26.153788 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:26.153789 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:26.153799 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:26.153844 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:28:26.153993 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:26.154740 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:26.155799 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:26.156102 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:26.156158 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:26.156321 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:28:26.156352 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:28:26.156472 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:26.156718 (+ 246us) server_negotiation.cc:299] Negotiation successful
0928 06:28:26.156737 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:28:26.162582 25064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:26.159465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:41705 (local address 127.22.225.129:43685)
0928 06:28:26.159532 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:26.159534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:26.159617 (+ 83us) server_negotiation.cc:407] Connection header received
0928 06:28:26.159679 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:26.159680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:26.159690 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:26.159747 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:28:26.159900 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:26.160635 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:26.161816 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:26.162104 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:26.162139 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:26.162355 (+ 216us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:28:26.162384 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:28:26.162410 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:26.162501 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 06:28:26.162517 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 06:28:26.162855 25104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:26.159407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.129:43685 (local address 127.22.225.129:41705)
0928 06:28:26.159554 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:26.159565 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:26.159627 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:26.159778 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:26.159778 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:26.159780 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:28:26.159836 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:26.159838 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:26.160739 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:26.160739 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:26.161705 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:26.161708 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:26.162197 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:26.162198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:26.162253 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:26.162269 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:28:26.162444 (+ 175us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:28:26.162445 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 06:28:26.162685 (+ 240us) client_negotiation.cc:240] Negotiation successful
0928 06:28:26.162756 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":83,"threads_started":1}
I20260928 06:28:26.167258 24984 raft_consensus.cc:1261] T 7c1e783f0fa44ab98db665a30353573e P d2bbaa906b854eaeb416b9b294bda97f [term 1 FOLLOWER]: Refusing update from remote peer 59eeadf53cb043659a02cba07a4300ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:28:26.167467 24844 raft_consensus.cc:1261] T 7c1e783f0fa44ab98db665a30353573e P c439d4c662d84a308470ceba08705718 [term 1 FOLLOWER]: Refusing update from remote peer 59eeadf53cb043659a02cba07a4300ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:28:26.167661 25061 consensus_queue.cc:1035] T 7c1e783f0fa44ab98db665a30353573e P 59eeadf53cb043659a02cba07a4300ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "c439d4c662d84a308470ceba08705718" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 39959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:28:26.167914 25061 consensus_queue.cc:1035] T 7c1e783f0fa44ab98db665a30353573e P 59eeadf53cb043659a02cba07a4300ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2bbaa906b854eaeb416b9b294bda97f" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:28:26.183355 24546 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37820:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:28:26.183457 24546 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37820:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:28:26.184237 24546 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4918f18b857f4b0696388f71b4355706: Sending DeleteTablet for 3 replicas of tablet 7c1e783f0fa44ab98db665a30353573e
I20260928 06:28:26.184501 24964 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c1e783f0fa44ab98db665a30353573e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:28:26 UTC) from {username='kudu', principal='kudu/127.22.225.190@KRBTEST.COM'} at 127.0.0.1:59810
I20260928 06:28:26.184620 25077 tablet_replica.cc:330] T 7c1e783f0fa44ab98db665a30353573e P d2bbaa906b854eaeb416b9b294bda97f: stopping tablet replica
I20260928 06:28:26.184633 24684 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c1e783f0fa44ab98db665a30353573e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:28:26 UTC) from {username='kudu', principal='kudu/127.22.225.190@KRBTEST.COM'} at 127.0.0.1:49218
I20260928 06:28:26.184626 24824 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c1e783f0fa44ab98db665a30353573e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:28:26 UTC) from {username='kudu', principal='kudu/127.22.225.190@KRBTEST.COM'} at 127.0.0.1:57648
I20260928 06:28:26.184671 25077 raft_consensus.cc:2229] T 7c1e783f0fa44ab98db665a30353573e P d2bbaa906b854eaeb416b9b294bda97f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:28:26.184702 25077 raft_consensus.cc:2258] T 7c1e783f0fa44ab98db665a30353573e P d2bbaa906b854eaeb416b9b294bda97f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:28:26.184718 25079 tablet_replica.cc:330] T 7c1e783f0fa44ab98db665a30353573e P c439d4c662d84a308470ceba08705718: stopping tablet replica
I20260928 06:28:26.184768 25079 raft_consensus.cc:2229] T 7c1e783f0fa44ab98db665a30353573e P c439d4c662d84a308470ceba08705718 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:28:26.184772 25078 tablet_replica.cc:330] T 7c1e783f0fa44ab98db665a30353573e P 59eeadf53cb043659a02cba07a4300ef: stopping tablet replica
I20260928 06:28:26.184805 25079 raft_consensus.cc:2258] T 7c1e783f0fa44ab98db665a30353573e P c439d4c662d84a308470ceba08705718 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:28:26.184828 25078 raft_consensus.cc:2229] T 7c1e783f0fa44ab98db665a30353573e P 59eeadf53cb043659a02cba07a4300ef [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:28:26.184900 25078 raft_consensus.cc:2258] T 7c1e783f0fa44ab98db665a30353573e P 59eeadf53cb043659a02cba07a4300ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:28:26.185248 25077 ts_tablet_manager.cc:1905] T 7c1e783f0fa44ab98db665a30353573e P d2bbaa906b854eaeb416b9b294bda97f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:28:26.185607 25079 ts_tablet_manager.cc:1905] T 7c1e783f0fa44ab98db665a30353573e P c439d4c662d84a308470ceba08705718: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:28:26.187009 25078 ts_tablet_manager.cc:1905] T 7c1e783f0fa44ab98db665a30353573e P 59eeadf53cb043659a02cba07a4300ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:28:26.187198 25079 ts_tablet_manager.cc:1918] T 7c1e783f0fa44ab98db665a30353573e P c439d4c662d84a308470ceba08705718: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:28:26.187170 25077 ts_tablet_manager.cc:1918] T 7c1e783f0fa44ab98db665a30353573e P d2bbaa906b854eaeb416b9b294bda97f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:28:26.187260 25079 log.cc:1196] T 7c1e783f0fa44ab98db665a30353573e P c439d4c662d84a308470ceba08705718: Deleting WAL directory at /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/ts-1/wal/wals/7c1e783f0fa44ab98db665a30353573e
I20260928 06:28:26.187260 25077 log.cc:1196] T 7c1e783f0fa44ab98db665a30353573e P d2bbaa906b854eaeb416b9b294bda97f: Deleting WAL directory at /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/ts-2/wal/wals/7c1e783f0fa44ab98db665a30353573e
I20260928 06:28:26.187493 25077 ts_tablet_manager.cc:1939] T 7c1e783f0fa44ab98db665a30353573e P d2bbaa906b854eaeb416b9b294bda97f: Deleting consensus metadata
I20260928 06:28:26.187493 25079 ts_tablet_manager.cc:1939] T 7c1e783f0fa44ab98db665a30353573e P c439d4c662d84a308470ceba08705718: Deleting consensus metadata
I20260928 06:28:26.187813 24531 catalog_manager.cc:4841] TS c439d4c662d84a308470ceba08705718 (127.22.225.130:39959): tablet 7c1e783f0fa44ab98db665a30353573e (table test-table [id=d20ac0b5aad947099555f2c0f674e503]) successfully deleted
I20260928 06:28:26.187901 24534 catalog_manager.cc:4841] TS d2bbaa906b854eaeb416b9b294bda97f (127.22.225.131:40979): tablet 7c1e783f0fa44ab98db665a30353573e (table test-table [id=d20ac0b5aad947099555f2c0f674e503]) successfully deleted
I20260928 06:28:26.188418 25078 ts_tablet_manager.cc:1918] T 7c1e783f0fa44ab98db665a30353573e P 59eeadf53cb043659a02cba07a4300ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:28:26.188493 25078 log.cc:1196] T 7c1e783f0fa44ab98db665a30353573e P 59eeadf53cb043659a02cba07a4300ef: Deleting WAL directory at /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576899143097-23430-0/minicluster-data/ts-0/wal/wals/7c1e783f0fa44ab98db665a30353573e
I20260928 06:28:26.188732 25078 ts_tablet_manager.cc:1939] T 7c1e783f0fa44ab98db665a30353573e P 59eeadf53cb043659a02cba07a4300ef: Deleting consensus metadata
I20260928 06:28:26.189020 24533 catalog_manager.cc:4841] TS 59eeadf53cb043659a02cba07a4300ef (127.22.225.129:43685): tablet 7c1e783f0fa44ab98db665a30353573e (table test-table [id=d20ac0b5aad947099555f2c0f674e503]) successfully deleted
Sep 28 06:28:26 dist-test-slave-d59m krb5kdc[24502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576905, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.225.129@KRBTEST.COM
I20260928 06:28:26.199333 25064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:26.189313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49238 (local address 127.22.225.129:43685)
0928 06:28:26.189352 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:26.189354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:26.189401 (+ 47us) server_negotiation.cc:407] Connection header received
0928 06:28:26.189471 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:26.189472 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:26.189481 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:26.189535 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:26.190197 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:26.190856 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:26.191904 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:26.192162 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:26.192226 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:26.195981 (+ 3755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:26.195990 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:26.195990 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:26.195998 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:26.198049 (+ 2051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:26.198472 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:26.198476 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:26.198476 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:26.198542 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:26.198743 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:26.198744 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:26.198744 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:26.198920 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:26.199058 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:26.199128 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 06:28:26.199218 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 06:28:26.199564 24744 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49238
I20260928 06:28:26.207755 24596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:26.200554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37842 (local address 127.22.225.190:46621)
0928 06:28:26.200621 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:26.200623 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:26.200818 (+ 195us) server_negotiation.cc:407] Connection header received
0928 06:28:26.200871 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:26.200872 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:26.200884 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:26.200942 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:26.201530 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:26.202207 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:26.203210 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:26.203478 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:26.203547 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:26.204957 (+ 1410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:26.204958 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:26.204959 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:26.204965 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:26.206590 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:26.206967 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:26.206968 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:26.206968 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:26.207028 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:26.207236 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:26.207237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:26.207237 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:26.207363 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:26.207463 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:26.207597 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 06:28:26.207670 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260928 06:28:26.208051 24546 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37842
I20260928 06:28:26.208729 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 24619
I20260928 06:28:26.217185 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 24759
I20260928 06:28:26.221907 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 24899
I20260928 06:28:26.229614 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 24515
2026-09-28T06:28:26Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2601 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:28:26 dist-test-slave-d59m krb5kdc[25116](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:28:26 dist-test-slave-d59m krb5kdc[25116](info): set up 2 sockets
Sep 28 06:28:26 dist-test-slave-d59m krb5kdc[25116](info): commencing operation
krb5kdc: starting...
W20260928 06:28:27.285661 23430 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Sep 28 06:28:27 dist-test-slave-d59m krb5kdc[25116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:28:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:28:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab.
Entry for principal kudu/127.22.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab.
I20260928 06:28:27.407680 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:36347
--webserver_interface=127.22.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab
--principal=kudu/127.22.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:34669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.225.190:36347
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:28:27.488126 25129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:28:27.488354 25129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:28:27.488394 25129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:28:27.489619 25129 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:28:27.489683 25129 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 06:28:27.489705 25129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:28:27.489727 25129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:28:27.489755 25129 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:28:27.489771 25129 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:28:27.491238 25129 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:34669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.225.190:36347
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab
--principal=kudu/127.22.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:36347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:28:27.491516 25129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:28:27.491786 25129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:28:27.496748 25137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:27.496707 25134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:27.496716 25135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:27.497170 25129 server_base.cc:1034] running on GCE node
I20260928 06:28:27.497422 25129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:28:27.497702 25129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:28:27.498869 25129 hybrid_clock.cc:648] HybridClock initialized: now 1790576907498850 us; error 37 us; skew 500 ppm
Sep 28 06:28:27 dist-test-slave-d59m krb5kdc[25116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576907, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:28:27.501446 25129 init.cc:377] Logged in from keytab as kudu/127.22.225.190@KRBTEST.COM (short username kudu)
I20260928 06:28:27.501863 25129 webserver.cc:460] Webserver started at http://127.22.225.190:43619/ using document root <none> and password file <none>
I20260928 06:28:27.502097 25129 fs_manager.cc:359] Metadata directory not provided
I20260928 06:28:27.502164 25129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:28:27.502321 25129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:28:27.503384 25129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/master-0/data/instance:
uuid: "9128f0ee0f504bed81b2d03646b98fd4"
format_stamp: "Formatted at 2026-09-28 06:28:27 on dist-test-slave-d59m"
server_key: "563083d05b707fd518ed5639954a599b"
server_key_iv: "9797902950bda19753bc207159d822c0"
server_key_version: "encryptionkey@0"
I20260928 06:28:27.503742 25129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/master-0/wal/instance:
uuid: "9128f0ee0f504bed81b2d03646b98fd4"
format_stamp: "Formatted at 2026-09-28 06:28:27 on dist-test-slave-d59m"
server_key: "563083d05b707fd518ed5639954a599b"
server_key_iv: "9797902950bda19753bc207159d822c0"
server_key_version: "encryptionkey@0"
I20260928 06:28:27.505165 25129 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:28:27.505950 25144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:27.506152 25129 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:28:27.506258 25129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/master-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/master-0/wal
uuid: "9128f0ee0f504bed81b2d03646b98fd4"
format_stamp: "Formatted at 2026-09-28 06:28:27 on dist-test-slave-d59m"
server_key: "563083d05b707fd518ed5639954a599b"
server_key_iv: "9797902950bda19753bc207159d822c0"
server_key_version: "encryptionkey@0"
I20260928 06:28:27.506328 25129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:28:27.522096 25129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:28:27.524341 25129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:28:27.524504 25129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:28:27.529312 25129 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.190:36347
I20260928 06:28:27.529371 25207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.190:36347 every 8 connection(s)
I20260928 06:28:27.529677 25129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
I20260928 06:28:27.530404 25208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:27.532428 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 25129
I20260928 06:28:27.532543 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/master-0/wal/instance
I20260928 06:28:27.532737 23430 external_mini_cluster.cc:1428] Setting key 7c1aa9fa715a55ff32c77c13bf6073b1
I20260928 06:28:27.533381 25208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9128f0ee0f504bed81b2d03646b98fd4: Bootstrap starting.
I20260928 06:28:27.534082 25208 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9128f0ee0f504bed81b2d03646b98fd4: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:27.534385 25208 log.cc:824] T 00000000000000000000000000000000 P 9128f0ee0f504bed81b2d03646b98fd4: Log is configured to *not* fsync() on all Append() calls
I20260928 06:28:27.535146 25208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9128f0ee0f504bed81b2d03646b98fd4: No bootstrap required, opened a new log
I20260928 06:28:27.537513 25208 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9128f0ee0f504bed81b2d03646b98fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9128f0ee0f504bed81b2d03646b98fd4" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36347 } }
I20260928 06:28:27.537693 25208 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9128f0ee0f504bed81b2d03646b98fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:27.537731 25208 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9128f0ee0f504bed81b2d03646b98fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9128f0ee0f504bed81b2d03646b98fd4, State: Initialized, Role: FOLLOWER
I20260928 06:28:27.537868 25208 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9128f0ee0f504bed81b2d03646b98fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9128f0ee0f504bed81b2d03646b98fd4" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36347 } }
I20260928 06:28:27.537945 25208 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9128f0ee0f504bed81b2d03646b98fd4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:28:27.537981 25208 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9128f0ee0f504bed81b2d03646b98fd4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:28:27.538023 25208 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9128f0ee0f504bed81b2d03646b98fd4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:27.538635 25208 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9128f0ee0f504bed81b2d03646b98fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9128f0ee0f504bed81b2d03646b98fd4" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36347 } }
I20260928 06:28:27.538733 25208 leader_election.cc:302] T 00000000000000000000000000000000 P 9128f0ee0f504bed81b2d03646b98fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9128f0ee0f504bed81b2d03646b98fd4; no voters:
I20260928 06:28:27.538882 25208 leader_election.cc:288] T 00000000000000000000000000000000 P 9128f0ee0f504bed81b2d03646b98fd4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:28:27.538938 25213 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9128f0ee0f504bed81b2d03646b98fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:28:27.539081 25213 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9128f0ee0f504bed81b2d03646b98fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 9128f0ee0f504bed81b2d03646b98fd4, State: Running, Role: LEADER
I20260928 06:28:27.539119 25208 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9128f0ee0f504bed81b2d03646b98fd4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:28:27.539176 25213 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9128f0ee0f504bed81b2d03646b98fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9128f0ee0f504bed81b2d03646b98fd4" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36347 } }
Sep 28 06:28:27 dist-test-slave-d59m krb5kdc[25116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:28:27.539953 25213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9128f0ee0f504bed81b2d03646b98fd4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9128f0ee0f504bed81b2d03646b98fd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9128f0ee0f504bed81b2d03646b98fd4" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36347 } } }
I20260928 06:28:27.540026 25213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9128f0ee0f504bed81b2d03646b98fd4 [sys.catalog]: This master's current role is: LEADER
I20260928 06:28:27.540172 25213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9128f0ee0f504bed81b2d03646b98fd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9128f0ee0f504bed81b2d03646b98fd4. Latest consensus state: current_term: 1 leader_uuid: "9128f0ee0f504bed81b2d03646b98fd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9128f0ee0f504bed81b2d03646b98fd4" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36347 } } }
I20260928 06:28:27.540226 25213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9128f0ee0f504bed81b2d03646b98fd4 [sys.catalog]: This master's current role is: LEADER
I20260928 06:28:27.540331 25224 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:28:27.540977 25224 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:28:27.542806 25224 catalog_manager.cc:1347] Generated new cluster ID: 46085167bd254bdda45165f7777409cd
I20260928 06:28:27.542861 25224 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:28:27.544122 25211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:27.533251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43798 (local address 127.22.225.190:36347)
0928 06:28:27.533486 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:27.533497 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:27.533533 (+ 36us) server_negotiation.cc:407] Connection header received
0928 06:28:27.533960 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:27.533967 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:27.534039 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:27.534202 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:27.535019 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:27.535926 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:27.536749 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:27.537189 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:27.537309 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:27.539903 (+ 2594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:27.539907 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:27.539907 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:27.539919 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:27.542141 (+ 2222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:27.542563 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:27.542565 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:27.542566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:27.542664 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:27.542868 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:27.542869 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:27.542869 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:27.543162 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:27.543281 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:27.543770 (+ 489us) server_negotiation.cc:299] Negotiation successful
0928 06:28:27.543895 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":89,"threads_started":1}
I20260928 06:28:27.552325 25224 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:28:27.552907 25224 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:28:27.559916 25224 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9128f0ee0f504bed81b2d03646b98fd4: Generated new TSK 0
I20260928 06:28:27.560093 25224 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.225.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab.
Entry for principal kudu/127.22.225.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab.
I20260928 06:28:27.584064 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.129:0
--local_ip_for_outbound_sockets=127.22.225.129
--webserver_interface=127.22.225.129
--webserver_port=0
--tserver_master_addrs=127.22.225.190:36347
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab
--principal=kudu/127.22.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:34669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:28:27.677728 25234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:28:27.678001 25234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:28:27.678042 25234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:28:27.679479 25234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:28:27.679531 25234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:28:27.679559 25234 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 06:28:27.679575 25234 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 06:28:27.679597 25234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.129
I20260928 06:28:27.681500 25234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:34669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab
--principal=kudu/127.22.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.225.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.225.190:36347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.129
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:28:27.681730 25234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:28:27.681982 25234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:28:27.687413 25240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:27.687680 25239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:27.687706 25242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:27.688024 25234 server_base.cc:1034] running on GCE node
I20260928 06:28:27.688216 25234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:28:27.688450 25234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:28:27.689620 25234 hybrid_clock.cc:648] HybridClock initialized: now 1790576907689599 us; error 37 us; skew 500 ppm
Sep 28 06:28:27 dist-test-slave-d59m krb5kdc[25116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576907, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:28:27.692309 25234 init.cc:377] Logged in from keytab as kudu/127.22.225.129@KRBTEST.COM (short username kudu)
I20260928 06:28:27.692844 25234 webserver.cc:460] Webserver started at http://127.22.225.129:39397/ using document root <none> and password file <none>
I20260928 06:28:27.693164 25234 fs_manager.cc:359] Metadata directory not provided
I20260928 06:28:27.693244 25234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:28:27.693378 25234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:28:27.694391 25234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/ts-0/data/instance:
uuid: "1f5037e17b564c78be31bd2014807c9e"
format_stamp: "Formatted at 2026-09-28 06:28:27 on dist-test-slave-d59m"
server_key: "ed6e3e43dfa7499aa6ea09147eafe9c9"
server_key_iv: "82c7a9af4eccc9b562d61832412b0751"
server_key_version: "encryptionkey@0"
I20260928 06:28:27.694770 25234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/ts-0/wal/instance:
uuid: "1f5037e17b564c78be31bd2014807c9e"
format_stamp: "Formatted at 2026-09-28 06:28:27 on dist-test-slave-d59m"
server_key: "ed6e3e43dfa7499aa6ea09147eafe9c9"
server_key_iv: "82c7a9af4eccc9b562d61832412b0751"
server_key_version: "encryptionkey@0"
I20260928 06:28:27.696126 25234 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:28:27.696950 25249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:27.697119 25234 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:28:27.697219 25234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/ts-0/wal
uuid: "1f5037e17b564c78be31bd2014807c9e"
format_stamp: "Formatted at 2026-09-28 06:28:27 on dist-test-slave-d59m"
server_key: "ed6e3e43dfa7499aa6ea09147eafe9c9"
server_key_iv: "82c7a9af4eccc9b562d61832412b0751"
server_key_version: "encryptionkey@0"
I20260928 06:28:27.697299 25234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:28:27.728489 25234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:28:27.731108 25234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:28:27.731302 25234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:28:27.731927 25234 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:28:27.731988 25234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:27.732023 25234 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:28:27.732048 25234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 06:28:27 dist-test-slave-d59m krb5kdc[25116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576907, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.129@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:28:27.739629 25234 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.129:38889
I20260928 06:28:27.739714 25364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.129:38889 every 8 connection(s)
I20260928 06:28:27.740073 25234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
I20260928 06:28:27.742852 25211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:27.732377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:44881 (local address 127.22.225.190:36347)
0928 06:28:27.732437 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:27.732440 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:27.732728 (+ 288us) server_negotiation.cc:407] Connection header received
0928 06:28:27.733408 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:27.733409 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:27.733422 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:27.733499 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:27.734253 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:27.735114 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:27.736888 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:27.737179 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:27.737253 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:27.739987 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:27.739988 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:27.739989 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:27.739996 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:27.741550 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:27.741993 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:27.741994 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:27.741994 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:27.742058 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:27.742313 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:27.742314 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:27.742314 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:27.742405 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:27.742450 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:27.742731 (+ 281us) server_negotiation.cc:299] Negotiation successful
0928 06:28:27.742795 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 06:28:27.743048 25258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:27.732383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36347 (local address 127.22.225.129:44881)
0928 06:28:27.732570 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:27.732638 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:27.733280 (+ 642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:27.733581 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:27.733585 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:27.733900 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:27.734161 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:27.734168 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:27.735240 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:27.735241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:27.736797 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:27.736801 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:27.737272 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:27.737273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:27.737438 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:27.738001 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:27.738011 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:27.739874 (+ 1863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:27.741629 (+ 1755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:27.741635 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:27.741637 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:27.741917 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:27.742120 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:27.742121 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:27.742121 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:27.742257 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:27.742461 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:27.742466 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:27.742610 (+ 144us) client_negotiation.cc:769] Sending connection context
0928 06:28:27.742713 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 06:28:27.742855 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":93,"threads_started":1}
I20260928 06:28:27.743491 25365 heartbeater.cc:347] Connected to a master server at 127.22.225.190:36347
I20260928 06:28:27.743577 25365 heartbeater.cc:464] Registering TS with master...
I20260928 06:28:27.743758 25365 heartbeater.cc:511] Master 127.22.225.190:36347 requested a full tablet report, sending...
I20260928 06:28:27.744264 25161 ts_manager.cc:194] Registered new tserver with Master: 1f5037e17b564c78be31bd2014807c9e (127.22.225.129:38889)
I20260928 06:28:27.745201 25161 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.225.129@KRBTEST.COM'} at 127.22.225.129:44881
I20260928 06:28:27.749739 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 25234
I20260928 06:28:27.749832 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/ts-0/wal/instance
I20260928 06:28:27.750006 23430 external_mini_cluster.cc:1428] Setting key c7441469f58d63b08cc0233e5485c3e3
I20260928 06:28:27.750252 25211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:27.744670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:55465 (local address 127.22.225.190:36347)
0928 06:28:27.744703 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:27.744705 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:27.744712 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:28:27.744767 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:27.744767 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:27.744776 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:27.744829 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:27.745147 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:27.745819 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:27.746543 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:27.746797 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:27.746867 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:27.747676 (+ 809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:27.747677 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:27.747677 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:27.747684 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:27.749079 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:27.749395 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:27.749396 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:27.749396 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:27.749453 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:27.749653 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:27.749654 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:27.749654 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:27.749767 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:27.749911 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:27.750084 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 06:28:27.750161 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 06:28:27.750265 25371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:27.744497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36347 (local address 127.22.225.129:55465)
0928 06:28:27.744660 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:27.744672 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:27.744719 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:27.744832 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:27.744832 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:27.745008 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:27.745076 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:27.745077 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:27.745955 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:27.745956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:27.746463 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:27.746466 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:27.746871 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:27.746872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:27.746927 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:27.747307 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:27.747311 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:27.747595 (+ 284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:27.749168 (+ 1573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:27.749169 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:27.749169 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:27.749329 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:27.749530 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:27.749531 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:27.749531 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:27.749579 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:27.749881 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:27.749882 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:27.749928 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 06:28:27.750025 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 06:28:27.750163 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":91,"threads_started":1}
WARNING: no policy specified for kudu/127.22.225.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab.
Entry for principal kudu/127.22.225.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab.
I20260928 06:28:27.774912 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.130:0
--local_ip_for_outbound_sockets=127.22.225.130
--webserver_interface=127.22.225.130
--webserver_port=0
--tserver_master_addrs=127.22.225.190:36347
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab
--principal=kudu/127.22.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:34669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:28:27.855450 25374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:28:27.855680 25374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:28:27.855721 25374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:28:27.856879 25374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:28:27.856945 25374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:28:27.856979 25374 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 06:28:27.857010 25374 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 06:28:27.857054 25374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.130
I20260928 06:28:27.858495 25374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:34669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab
--principal=kudu/127.22.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.225.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.225.190:36347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.130
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:28:27.858748 25374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:28:27.858973 25374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:28:27.863847 25379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:27.863933 25382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:27.863862 25380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:27.864571 25374 server_base.cc:1034] running on GCE node
I20260928 06:28:27.864801 25374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:28:27.865060 25374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:28:27.866214 25374 hybrid_clock.cc:648] HybridClock initialized: now 1790576907866201 us; error 29 us; skew 500 ppm
Sep 28 06:28:27 dist-test-slave-d59m krb5kdc[25116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576907, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:28:27.868942 25374 init.cc:377] Logged in from keytab as kudu/127.22.225.130@KRBTEST.COM (short username kudu)
I20260928 06:28:27.869508 25374 webserver.cc:460] Webserver started at http://127.22.225.130:43979/ using document root <none> and password file <none>
I20260928 06:28:27.869758 25374 fs_manager.cc:359] Metadata directory not provided
I20260928 06:28:27.869827 25374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:28:27.869967 25374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:28:27.871313 25374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/ts-1/data/instance:
uuid: "533e2ce30a5e48c6be272da7a20dfd63"
format_stamp: "Formatted at 2026-09-28 06:28:27 on dist-test-slave-d59m"
server_key: "2075b68bc5c183ea491a73f1d4db9c0d"
server_key_iv: "2993b56f92bf0820132513f747addbcf"
server_key_version: "encryptionkey@0"
I20260928 06:28:27.871726 25374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/ts-1/wal/instance:
uuid: "533e2ce30a5e48c6be272da7a20dfd63"
format_stamp: "Formatted at 2026-09-28 06:28:27 on dist-test-slave-d59m"
server_key: "2075b68bc5c183ea491a73f1d4db9c0d"
server_key_iv: "2993b56f92bf0820132513f747addbcf"
server_key_version: "encryptionkey@0"
I20260928 06:28:27.873476 25374 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260928 06:28:27.874339 25389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:27.874531 25374 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:28:27.874630 25374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/ts-1/wal
uuid: "533e2ce30a5e48c6be272da7a20dfd63"
format_stamp: "Formatted at 2026-09-28 06:28:27 on dist-test-slave-d59m"
server_key: "2075b68bc5c183ea491a73f1d4db9c0d"
server_key_iv: "2993b56f92bf0820132513f747addbcf"
server_key_version: "encryptionkey@0"
I20260928 06:28:27.874704 25374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:28:27.893963 25374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:28:27.896116 25374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:28:27.896262 25374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:28:27.896960 25374 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:28:27.897009 25374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:27.897049 25374 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:28:27.897069 25374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 06:28:27 dist-test-slave-d59m krb5kdc[25116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576907, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.130@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:28:27.904731 25374 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.130:46149
I20260928 06:28:27.904837 25504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.130:46149 every 8 connection(s)
I20260928 06:28:27.905130 25374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
I20260928 06:28:27.908110 25211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:27.897661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:35069 (local address 127.22.225.190:36347)
0928 06:28:27.897711 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:27.897713 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:27.897724 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:28:27.898339 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:27.898341 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:27.898355 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:27.898441 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:27.899272 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:27.900158 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:27.901861 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:27.902156 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:27.902231 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:27.905145 (+ 2914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:27.905147 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:27.905147 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:27.905154 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:27.906817 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:27.907196 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:27.907197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:27.907198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:27.907257 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:27.907489 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:27.907490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:27.907490 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:27.907632 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:27.907699 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:27.907941 (+ 242us) server_negotiation.cc:299] Negotiation successful
0928 06:28:27.908017 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 06:28:27.908208 25398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:27.897287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36347 (local address 127.22.225.130:35069)
0928 06:28:27.897502 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:27.897559 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:27.898214 (+ 655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:27.898530 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:27.898536 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:27.898891 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:27.899152 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:27.899159 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:27.900236 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:27.900237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:27.901776 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:27.901780 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:27.902252 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:27.902254 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:27.902411 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:27.902962 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:27.902981 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:27.905038 (+ 2057us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:27.906897 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:27.906903 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:27.906905 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:27.907123 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:27.907321 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:27.907322 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:27.907322 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:27.907415 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:27.907695 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:27.907699 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:27.907807 (+ 108us) client_negotiation.cc:769] Sending connection context
0928 06:28:27.907888 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 06:28:27.908023 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":132,"threads_started":1}
I20260928 06:28:27.909642 25505 heartbeater.cc:347] Connected to a master server at 127.22.225.190:36347
I20260928 06:28:27.909731 25505 heartbeater.cc:464] Registering TS with master...
I20260928 06:28:27.909802 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 25374
I20260928 06:28:27.909881 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/ts-1/wal/instance
I20260928 06:28:27.909902 25505 heartbeater.cc:511] Master 127.22.225.190:36347 requested a full tablet report, sending...
I20260928 06:28:27.910079 23430 external_mini_cluster.cc:1428] Setting key 0a5f9ca1efeba9c0633059dbfef1b627
I20260928 06:28:27.910431 25162 ts_manager.cc:194] Registered new tserver with Master: 533e2ce30a5e48c6be272da7a20dfd63 (127.22.225.130:46149)
I20260928 06:28:27.910897 25162 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.225.130@KRBTEST.COM'} at 127.22.225.130:35069
I20260928 06:28:27.917128 25211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:27.909696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:50103 (local address 127.22.225.190:36347)
0928 06:28:27.909732 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:27.909734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:27.909932 (+ 198us) server_negotiation.cc:407] Connection header received
0928 06:28:27.910001 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:27.910002 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:27.910011 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:27.910052 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:27.910711 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:27.911417 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:27.912536 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:27.912860 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:27.912938 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:27.914019 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:27.914021 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:27.914021 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:27.914028 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:27.915847 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:27.916244 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:27.916245 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:27.916245 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:27.916298 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:27.916478 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:27.916479 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:27.916479 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:27.916690 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:27.916735 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:27.916955 (+ 220us) server_negotiation.cc:299] Negotiation successful
0928 06:28:27.917033 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 06:28:27.917462 25511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:27.909761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36347 (local address 127.22.225.130:50103)
0928 06:28:27.909876 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:27.909886 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:27.909945 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:27.910087 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:27.910088 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:27.910286 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:27.910371 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:27.910372 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:27.911549 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:27.911549 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:27.912423 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:27.912426 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:27.912944 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:27.912945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:27.913023 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:27.913545 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:27.913550 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:27.913918 (+ 368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:27.915942 (+ 2024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:27.915943 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:27.915943 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:27.916160 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:27.916364 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:27.916364 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:27.916365 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:27.916416 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:27.916802 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:27.916802 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:27.916850 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 06:28:27.917256 (+ 406us) client_negotiation.cc:240] Negotiation successful
0928 06:28:27.917365 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":67,"threads_started":1}
WARNING: no policy specified for kudu/127.22.225.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab.
Entry for principal kudu/127.22.225.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab.
I20260928 06:28:27.936568 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.131:0
--local_ip_for_outbound_sockets=127.22.225.131
--webserver_interface=127.22.225.131
--webserver_port=0
--tserver_master_addrs=127.22.225.190:36347
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab
--principal=kudu/127.22.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:34669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:28:28.017472 25514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:28:28.017699 25514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:28:28.017742 25514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:28:28.018870 25514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:28:28.018942 25514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:28:28.018994 25514 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 06:28:28.019039 25514 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 06:28:28.019079 25514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.131
I20260928 06:28:28.020576 25514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:34669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab
--principal=kudu/127.22.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.225.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.225.190:36347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.131
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:28:28.020900 25514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:28:28.021127 25514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:28:28.026024 25520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:28.026063 25519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:28.026037 25522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:28.026474 25514 server_base.cc:1034] running on GCE node
I20260928 06:28:28.026698 25514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:28:28.026961 25514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:28:28.028148 25514 hybrid_clock.cc:648] HybridClock initialized: now 1790576908028134 us; error 27 us; skew 500 ppm
Sep 28 06:28:28 dist-test-slave-d59m krb5kdc[25116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576908, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:28:28.031080 25514 init.cc:377] Logged in from keytab as kudu/127.22.225.131@KRBTEST.COM (short username kudu)
I20260928 06:28:28.031605 25514 webserver.cc:460] Webserver started at http://127.22.225.131:45625/ using document root <none> and password file <none>
I20260928 06:28:28.031823 25514 fs_manager.cc:359] Metadata directory not provided
I20260928 06:28:28.031886 25514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:28:28.032039 25514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:28:28.033089 25514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/ts-2/data/instance:
uuid: "6f91fe4bd8464f3bad645b052ecd1147"
format_stamp: "Formatted at 2026-09-28 06:28:28 on dist-test-slave-d59m"
server_key: "1f5d65b05a9ddc4f839a001cb4f4e686"
server_key_iv: "41a720e19086076fa3c859153bac0f36"
server_key_version: "encryptionkey@0"
I20260928 06:28:28.033429 25514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/ts-2/wal/instance:
uuid: "6f91fe4bd8464f3bad645b052ecd1147"
format_stamp: "Formatted at 2026-09-28 06:28:28 on dist-test-slave-d59m"
server_key: "1f5d65b05a9ddc4f839a001cb4f4e686"
server_key_iv: "41a720e19086076fa3c859153bac0f36"
server_key_version: "encryptionkey@0"
I20260928 06:28:28.034881 25514 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:28:28.035777 25529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:28.035941 25514 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:28:28.036041 25514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/ts-2/wal
uuid: "6f91fe4bd8464f3bad645b052ecd1147"
format_stamp: "Formatted at 2026-09-28 06:28:28 on dist-test-slave-d59m"
server_key: "1f5d65b05a9ddc4f839a001cb4f4e686"
server_key_iv: "41a720e19086076fa3c859153bac0f36"
server_key_version: "encryptionkey@0"
I20260928 06:28:28.036113 25514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:28:28.054025 25514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:28:28.056104 25514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:28:28.056260 25514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:28:28.056919 25514 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:28:28.056982 25514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:28.057021 25514 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:28:28.057041 25514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 06:28:28 dist-test-slave-d59m krb5kdc[25116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576908, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.131@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:28:28.064973 25514 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.131:40825
I20260928 06:28:28.065384 25514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
I20260928 06:28:28.065619 25644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.131:40825 every 8 connection(s)
I20260928 06:28:28.067842 25211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:28.057324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:48983 (local address 127.22.225.190:36347)
0928 06:28:28.057370 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:28.057372 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:28.057634 (+ 262us) server_negotiation.cc:407] Connection header received
0928 06:28:28.058395 (+ 761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:28.058397 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:28.058413 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:28.058482 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:28.059241 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:28.060089 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:28.061596 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:28.061919 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:28.061985 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:28.064598 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:28.064600 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:28.064600 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:28.064608 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:28.066368 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:28.066852 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:28.066854 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:28.066854 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:28.066926 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:28.067186 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:28.067187 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:28.067187 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:28.067307 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:28.067382 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:28.067659 (+ 277us) server_negotiation.cc:299] Negotiation successful
0928 06:28:28.067754 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 06:28:28.068007 25538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:28.057331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36347 (local address 127.22.225.131:48983)
0928 06:28:28.057501 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:28.057550 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:28.058257 (+ 707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:28.058573 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:28.058577 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:28.058885 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:28.059136 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:28.059143 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:28.060200 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:28.060201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:28.061497 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:28.061500 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:28.062027 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:28.062028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:28.062174 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:28.062727 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:28.062737 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:28.064467 (+ 1730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:28.066468 (+ 2001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:28.066474 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:28.066476 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:28.066740 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:28.066994 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:28.066995 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:28.066996 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:28.067110 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:28.067379 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:28.067386 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:28.067521 (+ 135us) client_negotiation.cc:769] Sending connection context
0928 06:28:28.067624 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 06:28:28.067775 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":81,"threads_started":1}
I20260928 06:28:28.068555 25645 heartbeater.cc:347] Connected to a master server at 127.22.225.190:36347
I20260928 06:28:28.068651 25645 heartbeater.cc:464] Registering TS with master...
I20260928 06:28:28.068823 25645 heartbeater.cc:511] Master 127.22.225.190:36347 requested a full tablet report, sending...
I20260928 06:28:28.069171 25161 ts_manager.cc:194] Registered new tserver with Master: 6f91fe4bd8464f3bad645b052ecd1147 (127.22.225.131:40825)
I20260928 06:28:28.069567 25161 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.225.131@KRBTEST.COM'} at 127.22.225.131:48983
I20260928 06:28:28.071480 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 25514
I20260928 06:28:28.071564 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/ts-2/wal/instance
I20260928 06:28:28.071740 23430 external_mini_cluster.cc:1428] Setting key 35774f9a70b7f665a9b02a369edeccac
I20260928 06:28:28.072923 23430 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:28:28.076094 25211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:28.069711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:60041 (local address 127.22.225.190:36347)
0928 06:28:28.069951 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:28.069953 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:28.069961 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:28:28.069978 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:28.069979 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:28.069988 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:28.070036 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:28.070439 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:28.070875 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:28.071745 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:28.072052 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:28.072097 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:28.073332 (+ 1235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:28.073333 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:28.073333 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:28.073340 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:28.074995 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:28.075314 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:28.075315 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:28.075315 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:28.075366 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:28.075566 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:28.075567 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:28.075568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:28.075673 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:28.075743 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:28.075933 (+ 190us) server_negotiation.cc:299] Negotiation successful
0928 06:28:28.076005 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222}
I20260928 06:28:28.076112 25651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:28.069627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36347 (local address 127.22.225.131:60041)
0928 06:28:28.069753 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:28.069763 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:28.069806 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:28.070076 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:28.070077 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:28.070267 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:28.070333 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:28.070335 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:28.070967 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:28.070968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:28.071653 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:28.071656 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:28.072221 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:28.072222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:28.072300 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:28.072779 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:28.072785 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:28.073241 (+ 456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:28.075077 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:28.075077 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:28.075078 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:28.075243 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:28.075439 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:28.075440 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:28.075441 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:28.075496 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:28.075747 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:28.075748 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:28.075821 (+ 73us) client_negotiation.cc:769] Sending connection context
0928 06:28:28.075906 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 06:28:28.076007 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":65,"threads_started":1}
Sep 28 06:28:28 dist-test-slave-d59m krb5kdc[25116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 06:28:28 dist-test-slave-d59m krb5kdc[25116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:28:28.101468 25211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:28.093813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43804 (local address 127.22.225.190:36347)
0928 06:28:28.093863 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:28.093866 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:28.093875 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:28:28.093904 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:28.093905 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:28.093916 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:28.093986 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:28.094510 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:28.095369 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:28.096085 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:28.096430 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:28.096506 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:28.098921 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:28.098923 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:28.098924 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:28.098932 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:28.100385 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:28.100806 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:28.100807 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:28.100807 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:28.100857 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:28.101049 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:28.101049 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:28.101049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:28.101138 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:28.101209 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:28.101355 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 06:28:28.101407 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 06:28:28.103398 25161 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43804:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:28:28.105085 25161 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:28:28.112844 25662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:28.108100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.131:40825 (local address 127.0.0.1:43402)
0928 06:28:28.108873 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:28.108883 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:28.108936 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:28.109164 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:28.109165 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:28.109167 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:28:28.109210 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:28.109211 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:28.110571 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:28.110572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:28.111651 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:28.111654 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:28.112506 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:28.112507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:28.112650 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:28.112667 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 06:28:28.112715 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 06:28:28.112766 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":761,"mutex_wait_us":1,"thread_start_us":63,"threads_started":1}
I20260928 06:28:28.113215 25667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:28.108294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43402 (local address 127.22.225.131:40825)
0928 06:28:28.108633 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:28.108638 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:28.108954 (+ 316us) server_negotiation.cc:407] Connection header received
0928 06:28:28.109005 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:28.109007 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:28.109023 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:28.109100 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:28:28.109282 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:28.110474 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:28.111751 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:28.112396 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:28.112980 (+ 584us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:28.113046 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:28.113077 (+ 31us) server_negotiation.cc:299] Negotiation successful
0928 06:28:28.113116 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":156,"threads_started":1}
I20260928 06:28:28.113380 25663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:28.107852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.130:46149 (local address 127.0.0.1:37518)
0928 06:28:28.108186 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:28.108229 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:28.108297 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:28.109263 (+ 966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:28.109264 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:28.109266 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:28:28.109309 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:28.109310 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:28.110541 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:28.110542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:28.111769 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:28.111772 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:28.112761 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:28.112762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:28.112829 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:28.112834 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:28:28.112866 (+ 32us) client_negotiation.cc:240] Negotiation successful
0928 06:28:28.113293 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":29,"thread_start_us":53,"threads_started":1}
I20260928 06:28:28.113737 25665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:28.107866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.129:38889 (local address 127.0.0.1:56874)
0928 06:28:28.108304 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:28.108319 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:28.108364 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:28.108695 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:28.108699 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:28.108701 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:28:28.108786 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:28.108788 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:28.110156 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:28.110157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:28.111770 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:28.111772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:28.113047 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:28.113047 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:28.113583 (+ 536us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:28.113586 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:28:28.113638 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 06:28:28.113676 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"mutex_wait_us":479,"thread_start_us":93,"threads_started":1}
I20260928 06:28:28.113942 25664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:28.107970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56874 (local address 127.22.225.129:38889)
0928 06:28:28.108159 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:28.108166 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:28.108443 (+ 277us) server_negotiation.cc:407] Connection header received
0928 06:28:28.108491 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:28.108494 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:28.108511 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:28.108583 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:28:28.108859 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:28.110065 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:28.111873 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:28.112655 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:28.113097 (+ 442us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:28.113176 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:28.113657 (+ 481us) server_negotiation.cc:299] Negotiation successful
0928 06:28:28.113695 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":102,"threads_started":1}
I20260928 06:28:28.113864 25567 tablet_service.cc:1467] Processing CreateTablet for tablet 6308733bfdbd43a9a3bd5bf7ee5d33aa (DEFAULT_TABLE table=test-table [id=be48279697c247b996057cdac7559108]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:28:28.114161 25666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:28.108111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37518 (local address 127.22.225.130:46149)
0928 06:28:28.109049 (+ 938us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:28.109054 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:28.109063 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:28:28.109117 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:28.109119 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:28.109134 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:28.109377 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:28:28.109432 (+ 55us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:28.110430 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:28.111860 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:28.112659 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:28.113910 (+ 1251us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:28.113974 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:28.114002 (+ 28us) server_negotiation.cc:299] Negotiation successful
0928 06:28:28.114050 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":903,"thread_start_us":87,"threads_started":1}
I20260928 06:28:28.114220 25567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6308733bfdbd43a9a3bd5bf7ee5d33aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:28.114673 25299 tablet_service.cc:1467] Processing CreateTablet for tablet 6308733bfdbd43a9a3bd5bf7ee5d33aa (DEFAULT_TABLE table=test-table [id=be48279697c247b996057cdac7559108]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:28:28.114675 25432 tablet_service.cc:1467] Processing CreateTablet for tablet 6308733bfdbd43a9a3bd5bf7ee5d33aa (DEFAULT_TABLE table=test-table [id=be48279697c247b996057cdac7559108]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:28:28.114980 25432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6308733bfdbd43a9a3bd5bf7ee5d33aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:28.114980 25299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6308733bfdbd43a9a3bd5bf7ee5d33aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:28.117386 25668 tablet_bootstrap.cc:492] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 6f91fe4bd8464f3bad645b052ecd1147: Bootstrap starting.
I20260928 06:28:28.118219 25668 tablet_bootstrap.cc:654] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 6f91fe4bd8464f3bad645b052ecd1147: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:28.118472 25668 log.cc:824] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 6f91fe4bd8464f3bad645b052ecd1147: Log is configured to *not* fsync() on all Append() calls
I20260928 06:28:28.118669 25670 tablet_bootstrap.cc:492] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 1f5037e17b564c78be31bd2014807c9e: Bootstrap starting.
I20260928 06:28:28.119268 25670 tablet_bootstrap.cc:654] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 1f5037e17b564c78be31bd2014807c9e: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:28.119515 25668 tablet_bootstrap.cc:492] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 6f91fe4bd8464f3bad645b052ecd1147: No bootstrap required, opened a new log
I20260928 06:28:28.119594 25670 log.cc:824] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 1f5037e17b564c78be31bd2014807c9e: Log is configured to *not* fsync() on all Append() calls
I20260928 06:28:28.119607 25668 ts_tablet_manager.cc:1397] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 6f91fe4bd8464f3bad645b052ecd1147: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:28:28.119915 25669 tablet_bootstrap.cc:492] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 533e2ce30a5e48c6be272da7a20dfd63: Bootstrap starting.
I20260928 06:28:28.120374 25670 tablet_bootstrap.cc:492] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 1f5037e17b564c78be31bd2014807c9e: No bootstrap required, opened a new log
I20260928 06:28:28.120493 25669 tablet_bootstrap.cc:654] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 533e2ce30a5e48c6be272da7a20dfd63: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:28.120503 25670 ts_tablet_manager.cc:1397] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 1f5037e17b564c78be31bd2014807c9e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:28:28.120826 25669 log.cc:824] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 533e2ce30a5e48c6be272da7a20dfd63: Log is configured to *not* fsync() on all Append() calls
I20260928 06:28:28.121512 25669 tablet_bootstrap.cc:492] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 533e2ce30a5e48c6be272da7a20dfd63: No bootstrap required, opened a new log
I20260928 06:28:28.121579 25669 ts_tablet_manager.cc:1397] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 533e2ce30a5e48c6be272da7a20dfd63: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:28:28.121749 25668 raft_consensus.cc:348] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 6f91fe4bd8464f3bad645b052ecd1147 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5037e17b564c78be31bd2014807c9e" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 38889 } } peers { permanent_uuid: "533e2ce30a5e48c6be272da7a20dfd63" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46149 } } peers { permanent_uuid: "6f91fe4bd8464f3bad645b052ecd1147" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40825 } }
I20260928 06:28:28.121918 25668 raft_consensus.cc:374] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 6f91fe4bd8464f3bad645b052ecd1147 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:28.121961 25668 raft_consensus.cc:729] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 6f91fe4bd8464f3bad645b052ecd1147 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f91fe4bd8464f3bad645b052ecd1147, State: Initialized, Role: FOLLOWER
I20260928 06:28:28.122094 25668 consensus_queue.cc:260] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 6f91fe4bd8464f3bad645b052ecd1147 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5037e17b564c78be31bd2014807c9e" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 38889 } } peers { permanent_uuid: "533e2ce30a5e48c6be272da7a20dfd63" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46149 } } peers { permanent_uuid: "6f91fe4bd8464f3bad645b052ecd1147" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40825 } }
I20260928 06:28:28.122376 25668 ts_tablet_manager.cc:1428] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 6f91fe4bd8464f3bad645b052ecd1147: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:28:28.122400 25645 heartbeater.cc:503] Master 127.22.225.190:36347 was elected leader, sending a full tablet report...
I20260928 06:28:28.123381 25669 raft_consensus.cc:348] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 533e2ce30a5e48c6be272da7a20dfd63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5037e17b564c78be31bd2014807c9e" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 38889 } } peers { permanent_uuid: "533e2ce30a5e48c6be272da7a20dfd63" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46149 } } peers { permanent_uuid: "6f91fe4bd8464f3bad645b052ecd1147" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40825 } }
I20260928 06:28:28.123550 25669 raft_consensus.cc:374] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 533e2ce30a5e48c6be272da7a20dfd63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:28.123587 25669 raft_consensus.cc:729] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 533e2ce30a5e48c6be272da7a20dfd63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 533e2ce30a5e48c6be272da7a20dfd63, State: Initialized, Role: FOLLOWER
I20260928 06:28:28.123682 25669 consensus_queue.cc:260] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 533e2ce30a5e48c6be272da7a20dfd63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5037e17b564c78be31bd2014807c9e" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 38889 } } peers { permanent_uuid: "533e2ce30a5e48c6be272da7a20dfd63" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46149 } } peers { permanent_uuid: "6f91fe4bd8464f3bad645b052ecd1147" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40825 } }
I20260928 06:28:28.123904 25669 ts_tablet_manager.cc:1428] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 533e2ce30a5e48c6be272da7a20dfd63: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 06:28:28.123951 25505 heartbeater.cc:503] Master 127.22.225.190:36347 was elected leader, sending a full tablet report...
I20260928 06:28:28.123906 25670 raft_consensus.cc:348] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 1f5037e17b564c78be31bd2014807c9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5037e17b564c78be31bd2014807c9e" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 38889 } } peers { permanent_uuid: "533e2ce30a5e48c6be272da7a20dfd63" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46149 } } peers { permanent_uuid: "6f91fe4bd8464f3bad645b052ecd1147" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40825 } }
I20260928 06:28:28.124091 25670 raft_consensus.cc:374] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 1f5037e17b564c78be31bd2014807c9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:28.124123 25670 raft_consensus.cc:729] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 1f5037e17b564c78be31bd2014807c9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f5037e17b564c78be31bd2014807c9e, State: Initialized, Role: FOLLOWER
I20260928 06:28:28.124235 25670 consensus_queue.cc:260] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 1f5037e17b564c78be31bd2014807c9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5037e17b564c78be31bd2014807c9e" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 38889 } } peers { permanent_uuid: "533e2ce30a5e48c6be272da7a20dfd63" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46149 } } peers { permanent_uuid: "6f91fe4bd8464f3bad645b052ecd1147" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40825 } }
I20260928 06:28:28.124470 25670 ts_tablet_manager.cc:1428] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 1f5037e17b564c78be31bd2014807c9e: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260928 06:28:28.124624 25365 heartbeater.cc:503] Master 127.22.225.190:36347 was elected leader, sending a full tablet report...
W20260928 06:28:28.155977 25506 tablet.cc:2369] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 533e2ce30a5e48c6be272da7a20dfd63: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:28:28.241274 25366 tablet.cc:2369] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 1f5037e17b564c78be31bd2014807c9e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:28:28.316841 25646 tablet.cc:2369] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 6f91fe4bd8464f3bad645b052ecd1147: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:28:28.393712 25675 raft_consensus.cc:482] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 533e2ce30a5e48c6be272da7a20dfd63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:28:28.393867 25675 raft_consensus.cc:504] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 533e2ce30a5e48c6be272da7a20dfd63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5037e17b564c78be31bd2014807c9e" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 38889 } } peers { permanent_uuid: "533e2ce30a5e48c6be272da7a20dfd63" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46149 } } peers { permanent_uuid: "6f91fe4bd8464f3bad645b052ecd1147" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40825 } }
I20260928 06:28:28.394245 25675 leader_election.cc:288] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 533e2ce30a5e48c6be272da7a20dfd63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f5037e17b564c78be31bd2014807c9e (127.22.225.129:38889), 6f91fe4bd8464f3bad645b052ecd1147 (127.22.225.131:40825)
I20260928 06:28:28.397869 25667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:28.394726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:49235 (local address 127.22.225.131:40825)
0928 06:28:28.394785 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:28.394787 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:28.394880 (+ 93us) server_negotiation.cc:407] Connection header received
0928 06:28:28.394951 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:28.394952 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:28.394962 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:28.395012 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:28:28.395153 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:28.395890 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:28.397121 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:28.397491 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:28.397577 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:28.397587 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:28.397732 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 06:28:28.397794 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 06:28:28.397883 25677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:28.394597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.131:40825 (local address 127.22.225.130:49235)
0928 06:28:28.394806 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:28.394815 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:28.394897 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:28.395014 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:28.395014 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:28.395016 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:28:28.395093 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:28.395095 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:28.395986 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:28.395986 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:28.397044 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:28.397045 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:28.397575 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:28.397576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:28.397648 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:28.397655 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 06:28:28.397726 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 06:28:28.397767 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":62,"threads_started":1}
I20260928 06:28:28.398029 25664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:28.394541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:33945 (local address 127.22.225.129:38889)
0928 06:28:28.394614 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:28.394617 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:28.394830 (+ 213us) server_negotiation.cc:407] Connection header received
0928 06:28:28.394860 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:28.394861 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:28.394871 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:28.394915 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:28:28.395107 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:28.395604 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:28.396945 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:28.397503 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:28.397590 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:28.397599 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:28.397912 (+ 313us) server_negotiation.cc:299] Negotiation successful
0928 06:28:28.397956 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 06:28:28.398140 25398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:28.394473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.129:38889 (local address 127.22.225.130:33945)
0928 06:28:28.394694 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:28.394706 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:28.394772 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:28.394961 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:28.394963 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:28.394964 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:28:28.395041 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:28.395043 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:28.395692 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:28.395693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:28.396855 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:28.396858 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:28.397633 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:28.397634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:28.397725 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:28.397728 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:28:28.397776 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 06:28:28.397816 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200}
I20260928 06:28:28.398411 25599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6308733bfdbd43a9a3bd5bf7ee5d33aa" candidate_uuid: "533e2ce30a5e48c6be272da7a20dfd63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f91fe4bd8464f3bad645b052ecd1147" is_pre_election: true
I20260928 06:28:28.398437 25319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6308733bfdbd43a9a3bd5bf7ee5d33aa" candidate_uuid: "533e2ce30a5e48c6be272da7a20dfd63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f5037e17b564c78be31bd2014807c9e" is_pre_election: true
I20260928 06:28:28.398574 25599 raft_consensus.cc:2454] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 6f91fe4bd8464f3bad645b052ecd1147 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 533e2ce30a5e48c6be272da7a20dfd63 in term 0.
I20260928 06:28:28.398574 25319 raft_consensus.cc:2454] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 1f5037e17b564c78be31bd2014807c9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 533e2ce30a5e48c6be272da7a20dfd63 in term 0.
I20260928 06:28:28.398797 25390 leader_election.cc:302] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 533e2ce30a5e48c6be272da7a20dfd63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 533e2ce30a5e48c6be272da7a20dfd63, 6f91fe4bd8464f3bad645b052ecd1147; no voters:
I20260928 06:28:28.398916 25675 raft_consensus.cc:2781] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 533e2ce30a5e48c6be272da7a20dfd63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:28:28.398996 25675 raft_consensus.cc:482] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 533e2ce30a5e48c6be272da7a20dfd63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:28:28.399020 25675 raft_consensus.cc:3037] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 533e2ce30a5e48c6be272da7a20dfd63 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:28.399981 25675 raft_consensus.cc:504] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 533e2ce30a5e48c6be272da7a20dfd63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5037e17b564c78be31bd2014807c9e" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 38889 } } peers { permanent_uuid: "533e2ce30a5e48c6be272da7a20dfd63" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46149 } } peers { permanent_uuid: "6f91fe4bd8464f3bad645b052ecd1147" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40825 } }
I20260928 06:28:28.400208 25675 leader_election.cc:288] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 533e2ce30a5e48c6be272da7a20dfd63 [CANDIDATE]: Term 1 election: Requested vote from peers 1f5037e17b564c78be31bd2014807c9e (127.22.225.129:38889), 6f91fe4bd8464f3bad645b052ecd1147 (127.22.225.131:40825)
I20260928 06:28:28.400264 25319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6308733bfdbd43a9a3bd5bf7ee5d33aa" candidate_uuid: "533e2ce30a5e48c6be272da7a20dfd63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f5037e17b564c78be31bd2014807c9e"
I20260928 06:28:28.400342 25319 raft_consensus.cc:3037] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 1f5037e17b564c78be31bd2014807c9e [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:28.400375 25599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6308733bfdbd43a9a3bd5bf7ee5d33aa" candidate_uuid: "533e2ce30a5e48c6be272da7a20dfd63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f91fe4bd8464f3bad645b052ecd1147"
I20260928 06:28:28.400439 25599 raft_consensus.cc:3037] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 6f91fe4bd8464f3bad645b052ecd1147 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:28.401309 25319 raft_consensus.cc:2454] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 1f5037e17b564c78be31bd2014807c9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 533e2ce30a5e48c6be272da7a20dfd63 in term 1.
I20260928 06:28:28.401340 25599 raft_consensus.cc:2454] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 6f91fe4bd8464f3bad645b052ecd1147 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 533e2ce30a5e48c6be272da7a20dfd63 in term 1.
I20260928 06:28:28.401484 25390 leader_election.cc:302] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 533e2ce30a5e48c6be272da7a20dfd63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f5037e17b564c78be31bd2014807c9e, 533e2ce30a5e48c6be272da7a20dfd63; no voters:
I20260928 06:28:28.401598 25675 raft_consensus.cc:2781] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 533e2ce30a5e48c6be272da7a20dfd63 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:28:28.401785 25675 raft_consensus.cc:686] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 533e2ce30a5e48c6be272da7a20dfd63 [term 1 LEADER]: Becoming Leader. State: Replica: 533e2ce30a5e48c6be272da7a20dfd63, State: Running, Role: LEADER
I20260928 06:28:28.401888 25675 consensus_queue.cc:237] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 533e2ce30a5e48c6be272da7a20dfd63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5037e17b564c78be31bd2014807c9e" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 38889 } } peers { permanent_uuid: "533e2ce30a5e48c6be272da7a20dfd63" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46149 } } peers { permanent_uuid: "6f91fe4bd8464f3bad645b052ecd1147" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40825 } }
I20260928 06:28:28.402714 25162 catalog_manager.cc:5495] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 533e2ce30a5e48c6be272da7a20dfd63 reported cstate change: term changed from 0 to 1, leader changed from <none> to 533e2ce30a5e48c6be272da7a20dfd63 (127.22.225.130). New cstate: current_term: 1 leader_uuid: "533e2ce30a5e48c6be272da7a20dfd63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5037e17b564c78be31bd2014807c9e" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 38889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "533e2ce30a5e48c6be272da7a20dfd63" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f91fe4bd8464f3bad645b052ecd1147" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40825 } health_report { overall_health: UNKNOWN } } }
Sep 28 06:28:28 dist-test-slave-d59m krb5kdc[25116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576907, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.190@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:28:28.497010 25211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:28.489434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43812 (local address 127.22.225.190:36347)
0928 06:28:28.489505 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:28.489507 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:28.489641 (+ 134us) server_negotiation.cc:407] Connection header received
0928 06:28:28.489719 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:28.489720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:28.489730 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:28.489800 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:28.490272 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:28.490789 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:28.491478 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:28.491661 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:28.491744 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:28.494304 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:28.494305 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:28.494305 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:28.494311 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:28.495691 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:28.496162 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:28.496163 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:28.496163 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:28.496211 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:28.496419 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:28.496419 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:28.496420 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:28.496515 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:28.496565 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:28.496775 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 06:28:28.496911 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":28}
I20260928 06:28:28.497020 25685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:28.489378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36347 (local address 127.0.0.1:43812)
0928 06:28:28.489559 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:28.489568 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:28.489661 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:28.489805 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:28.489806 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:28.490131 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:28.490178 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:28.490179 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:28.490868 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:28.490869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:28.491387 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:28.491389 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:28.491744 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:28.491745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:28.491800 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:28.492447 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:28.492452 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:28.494218 (+ 1766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:28.495798 (+ 1580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:28.495805 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:28.495808 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:28.496089 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:28.496277 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:28.496278 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:28.496278 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:28.496355 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:28.496599 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:28.496606 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:28.496664 (+ 58us) client_negotiation.cc:769] Sending connection context
0928 06:28:28.496738 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 06:28:28.496866 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":108,"threads_started":1}
I20260928 06:28:28.499239 25162 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.22.225.190@KRBTEST.COM'} at 127.0.0.1:43812:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 06:28:28.499471 25162 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:28:28.501019 25567 tablet_service.cc:1467] Processing CreateTablet for tablet 232ecec604ff4524af45471b49b08790 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=760c6223670b42b8952dfb004d243f6e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 06:28:28.501061 25299 tablet_service.cc:1467] Processing CreateTablet for tablet 232ecec604ff4524af45471b49b08790 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=760c6223670b42b8952dfb004d243f6e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 06:28:28.501075 25432 tablet_service.cc:1467] Processing CreateTablet for tablet 232ecec604ff4524af45471b49b08790 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=760c6223670b42b8952dfb004d243f6e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 06:28:28.501240 25432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 232ecec604ff4524af45471b49b08790. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:28.501240 25299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 232ecec604ff4524af45471b49b08790. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:28.501240 25567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 232ecec604ff4524af45471b49b08790. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:28.502389 25668 tablet_bootstrap.cc:492] T 232ecec604ff4524af45471b49b08790 P 6f91fe4bd8464f3bad645b052ecd1147: Bootstrap starting.
I20260928 06:28:28.502785 25670 tablet_bootstrap.cc:492] T 232ecec604ff4524af45471b49b08790 P 1f5037e17b564c78be31bd2014807c9e: Bootstrap starting.
I20260928 06:28:28.503000 25668 tablet_bootstrap.cc:654] T 232ecec604ff4524af45471b49b08790 P 6f91fe4bd8464f3bad645b052ecd1147: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:28.503175 25670 tablet_bootstrap.cc:654] T 232ecec604ff4524af45471b49b08790 P 1f5037e17b564c78be31bd2014807c9e: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:28.503479 25669 tablet_bootstrap.cc:492] T 232ecec604ff4524af45471b49b08790 P 533e2ce30a5e48c6be272da7a20dfd63: Bootstrap starting.
I20260928 06:28:28.503849 25669 tablet_bootstrap.cc:654] T 232ecec604ff4524af45471b49b08790 P 533e2ce30a5e48c6be272da7a20dfd63: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:28.503857 25668 tablet_bootstrap.cc:492] T 232ecec604ff4524af45471b49b08790 P 6f91fe4bd8464f3bad645b052ecd1147: No bootstrap required, opened a new log
I20260928 06:28:28.503914 25668 ts_tablet_manager.cc:1397] T 232ecec604ff4524af45471b49b08790 P 6f91fe4bd8464f3bad645b052ecd1147: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:28:28.504081 25668 raft_consensus.cc:348] T 232ecec604ff4524af45471b49b08790 P 6f91fe4bd8464f3bad645b052ecd1147 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5037e17b564c78be31bd2014807c9e" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 38889 } } peers { permanent_uuid: "533e2ce30a5e48c6be272da7a20dfd63" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46149 } } peers { permanent_uuid: "6f91fe4bd8464f3bad645b052ecd1147" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40825 } }
I20260928 06:28:28.504189 25668 raft_consensus.cc:374] T 232ecec604ff4524af45471b49b08790 P 6f91fe4bd8464f3bad645b052ecd1147 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:28.504211 25668 raft_consensus.cc:729] T 232ecec604ff4524af45471b49b08790 P 6f91fe4bd8464f3bad645b052ecd1147 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f91fe4bd8464f3bad645b052ecd1147, State: Initialized, Role: FOLLOWER
I20260928 06:28:28.504262 25668 consensus_queue.cc:260] T 232ecec604ff4524af45471b49b08790 P 6f91fe4bd8464f3bad645b052ecd1147 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5037e17b564c78be31bd2014807c9e" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 38889 } } peers { permanent_uuid: "533e2ce30a5e48c6be272da7a20dfd63" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46149 } } peers { permanent_uuid: "6f91fe4bd8464f3bad645b052ecd1147" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40825 } }
I20260928 06:28:28.504377 25668 ts_tablet_manager.cc:1428] T 232ecec604ff4524af45471b49b08790 P 6f91fe4bd8464f3bad645b052ecd1147: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 06:28:28.504513 25674 tablet_replica.cc:439] T 232ecec604ff4524af45471b49b08790 P 6f91fe4bd8464f3bad645b052ecd1147: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5037e17b564c78be31bd2014807c9e" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 38889 } } peers { permanent_uuid: "533e2ce30a5e48c6be272da7a20dfd63" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46149 } } peers { permanent_uuid: "6f91fe4bd8464f3bad645b052ecd1147" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40825 } } }
I20260928 06:28:28.504657 25674 tablet_replica.cc:442] T 232ecec604ff4524af45471b49b08790 P 6f91fe4bd8464f3bad645b052ecd1147: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:28:28.504931 25670 tablet_bootstrap.cc:492] T 232ecec604ff4524af45471b49b08790 P 1f5037e17b564c78be31bd2014807c9e: No bootstrap required, opened a new log
I20260928 06:28:28.504981 25670 ts_tablet_manager.cc:1397] T 232ecec604ff4524af45471b49b08790 P 1f5037e17b564c78be31bd2014807c9e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 06:28:28.505138 25670 raft_consensus.cc:348] T 232ecec604ff4524af45471b49b08790 P 1f5037e17b564c78be31bd2014807c9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5037e17b564c78be31bd2014807c9e" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 38889 } } peers { permanent_uuid: "533e2ce30a5e48c6be272da7a20dfd63" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46149 } } peers { permanent_uuid: "6f91fe4bd8464f3bad645b052ecd1147" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40825 } }
I20260928 06:28:28.505234 25670 raft_consensus.cc:374] T 232ecec604ff4524af45471b49b08790 P 1f5037e17b564c78be31bd2014807c9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:28.505254 25670 raft_consensus.cc:729] T 232ecec604ff4524af45471b49b08790 P 1f5037e17b564c78be31bd2014807c9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f5037e17b564c78be31bd2014807c9e, State: Initialized, Role: FOLLOWER
I20260928 06:28:28.505308 25670 consensus_queue.cc:260] T 232ecec604ff4524af45471b49b08790 P 1f5037e17b564c78be31bd2014807c9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5037e17b564c78be31bd2014807c9e" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 38889 } } peers { permanent_uuid: "533e2ce30a5e48c6be272da7a20dfd63" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46149 } } peers { permanent_uuid: "6f91fe4bd8464f3bad645b052ecd1147" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40825 } }
I20260928 06:28:28.505439 25670 ts_tablet_manager.cc:1428] T 232ecec604ff4524af45471b49b08790 P 1f5037e17b564c78be31bd2014807c9e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:28.505501 25676 tablet_replica.cc:439] T 232ecec604ff4524af45471b49b08790 P 1f5037e17b564c78be31bd2014807c9e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5037e17b564c78be31bd2014807c9e" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 38889 } } peers { permanent_uuid: "533e2ce30a5e48c6be272da7a20dfd63" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46149 } } peers { permanent_uuid: "6f91fe4bd8464f3bad645b052ecd1147" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40825 } } }
I20260928 06:28:28.505618 25676 tablet_replica.cc:442] T 232ecec604ff4524af45471b49b08790 P 1f5037e17b564c78be31bd2014807c9e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:28:28.505661 25669 tablet_bootstrap.cc:492] T 232ecec604ff4524af45471b49b08790 P 533e2ce30a5e48c6be272da7a20dfd63: No bootstrap required, opened a new log
I20260928 06:28:28.505716 25669 ts_tablet_manager.cc:1397] T 232ecec604ff4524af45471b49b08790 P 533e2ce30a5e48c6be272da7a20dfd63: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:28:28.505868 25669 raft_consensus.cc:348] T 232ecec604ff4524af45471b49b08790 P 533e2ce30a5e48c6be272da7a20dfd63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5037e17b564c78be31bd2014807c9e" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 38889 } } peers { permanent_uuid: "533e2ce30a5e48c6be272da7a20dfd63" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46149 } } peers { permanent_uuid: "6f91fe4bd8464f3bad645b052ecd1147" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40825 } }
I20260928 06:28:28.505923 25669 raft_consensus.cc:374] T 232ecec604ff4524af45471b49b08790 P 533e2ce30a5e48c6be272da7a20dfd63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:28.505936 25669 raft_consensus.cc:729] T 232ecec604ff4524af45471b49b08790 P 533e2ce30a5e48c6be272da7a20dfd63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 533e2ce30a5e48c6be272da7a20dfd63, State: Initialized, Role: FOLLOWER
I20260928 06:28:28.505990 25669 consensus_queue.cc:260] T 232ecec604ff4524af45471b49b08790 P 533e2ce30a5e48c6be272da7a20dfd63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5037e17b564c78be31bd2014807c9e" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 38889 } } peers { permanent_uuid: "533e2ce30a5e48c6be272da7a20dfd63" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46149 } } peers { permanent_uuid: "6f91fe4bd8464f3bad645b052ecd1147" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40825 } }
I20260928 06:28:28.506081 25669 ts_tablet_manager.cc:1428] T 232ecec604ff4524af45471b49b08790 P 533e2ce30a5e48c6be272da7a20dfd63: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:28.506131 25675 tablet_replica.cc:439] T 232ecec604ff4524af45471b49b08790 P 533e2ce30a5e48c6be272da7a20dfd63: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5037e17b564c78be31bd2014807c9e" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 38889 } } peers { permanent_uuid: "533e2ce30a5e48c6be272da7a20dfd63" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46149 } } peers { permanent_uuid: "6f91fe4bd8464f3bad645b052ecd1147" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40825 } } }
I20260928 06:28:28.506206 25675 tablet_replica.cc:442] T 232ecec604ff4524af45471b49b08790 P 533e2ce30a5e48c6be272da7a20dfd63: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:28:28.753878 25675 raft_consensus.cc:482] T 232ecec604ff4524af45471b49b08790 P 533e2ce30a5e48c6be272da7a20dfd63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:28:28.753973 25675 raft_consensus.cc:504] T 232ecec604ff4524af45471b49b08790 P 533e2ce30a5e48c6be272da7a20dfd63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5037e17b564c78be31bd2014807c9e" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 38889 } } peers { permanent_uuid: "533e2ce30a5e48c6be272da7a20dfd63" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46149 } } peers { permanent_uuid: "6f91fe4bd8464f3bad645b052ecd1147" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40825 } }
I20260928 06:28:28.754138 25675 leader_election.cc:288] T 232ecec604ff4524af45471b49b08790 P 533e2ce30a5e48c6be272da7a20dfd63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f5037e17b564c78be31bd2014807c9e (127.22.225.129:38889), 6f91fe4bd8464f3bad645b052ecd1147 (127.22.225.131:40825)
I20260928 06:28:28.754390 25599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232ecec604ff4524af45471b49b08790" candidate_uuid: "533e2ce30a5e48c6be272da7a20dfd63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f91fe4bd8464f3bad645b052ecd1147" is_pre_election: true
I20260928 06:28:28.754410 25319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232ecec604ff4524af45471b49b08790" candidate_uuid: "533e2ce30a5e48c6be272da7a20dfd63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f5037e17b564c78be31bd2014807c9e" is_pre_election: true
I20260928 06:28:28.754498 25599 raft_consensus.cc:2454] T 232ecec604ff4524af45471b49b08790 P 6f91fe4bd8464f3bad645b052ecd1147 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 533e2ce30a5e48c6be272da7a20dfd63 in term 0.
I20260928 06:28:28.754509 25319 raft_consensus.cc:2454] T 232ecec604ff4524af45471b49b08790 P 1f5037e17b564c78be31bd2014807c9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 533e2ce30a5e48c6be272da7a20dfd63 in term 0.
I20260928 06:28:28.754653 25390 leader_election.cc:302] T 232ecec604ff4524af45471b49b08790 P 533e2ce30a5e48c6be272da7a20dfd63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f5037e17b564c78be31bd2014807c9e, 533e2ce30a5e48c6be272da7a20dfd63; no voters:
I20260928 06:28:28.754760 25675 raft_consensus.cc:2781] T 232ecec604ff4524af45471b49b08790 P 533e2ce30a5e48c6be272da7a20dfd63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:28:28.754808 25675 raft_consensus.cc:482] T 232ecec604ff4524af45471b49b08790 P 533e2ce30a5e48c6be272da7a20dfd63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:28:28.754839 25675 raft_consensus.cc:3037] T 232ecec604ff4524af45471b49b08790 P 533e2ce30a5e48c6be272da7a20dfd63 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:28.755475 25675 raft_consensus.cc:504] T 232ecec604ff4524af45471b49b08790 P 533e2ce30a5e48c6be272da7a20dfd63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5037e17b564c78be31bd2014807c9e" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 38889 } } peers { permanent_uuid: "533e2ce30a5e48c6be272da7a20dfd63" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46149 } } peers { permanent_uuid: "6f91fe4bd8464f3bad645b052ecd1147" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40825 } }
I20260928 06:28:28.755602 25675 leader_election.cc:288] T 232ecec604ff4524af45471b49b08790 P 533e2ce30a5e48c6be272da7a20dfd63 [CANDIDATE]: Term 1 election: Requested vote from peers 1f5037e17b564c78be31bd2014807c9e (127.22.225.129:38889), 6f91fe4bd8464f3bad645b052ecd1147 (127.22.225.131:40825)
I20260928 06:28:28.755772 25319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232ecec604ff4524af45471b49b08790" candidate_uuid: "533e2ce30a5e48c6be272da7a20dfd63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f5037e17b564c78be31bd2014807c9e"
I20260928 06:28:28.755815 25599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232ecec604ff4524af45471b49b08790" candidate_uuid: "533e2ce30a5e48c6be272da7a20dfd63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f91fe4bd8464f3bad645b052ecd1147"
I20260928 06:28:28.755878 25319 raft_consensus.cc:3037] T 232ecec604ff4524af45471b49b08790 P 1f5037e17b564c78be31bd2014807c9e [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:28.755903 25599 raft_consensus.cc:3037] T 232ecec604ff4524af45471b49b08790 P 6f91fe4bd8464f3bad645b052ecd1147 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:28.756513 25599 raft_consensus.cc:2454] T 232ecec604ff4524af45471b49b08790 P 6f91fe4bd8464f3bad645b052ecd1147 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 533e2ce30a5e48c6be272da7a20dfd63 in term 1.
I20260928 06:28:28.756698 25390 leader_election.cc:302] T 232ecec604ff4524af45471b49b08790 P 533e2ce30a5e48c6be272da7a20dfd63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 533e2ce30a5e48c6be272da7a20dfd63, 6f91fe4bd8464f3bad645b052ecd1147; no voters:
I20260928 06:28:28.756699 25319 raft_consensus.cc:2454] T 232ecec604ff4524af45471b49b08790 P 1f5037e17b564c78be31bd2014807c9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 533e2ce30a5e48c6be272da7a20dfd63 in term 1.
I20260928 06:28:28.756784 25675 raft_consensus.cc:2781] T 232ecec604ff4524af45471b49b08790 P 533e2ce30a5e48c6be272da7a20dfd63 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:28:28.756842 25675 raft_consensus.cc:686] T 232ecec604ff4524af45471b49b08790 P 533e2ce30a5e48c6be272da7a20dfd63 [term 1 LEADER]: Becoming Leader. State: Replica: 533e2ce30a5e48c6be272da7a20dfd63, State: Running, Role: LEADER
I20260928 06:28:28.756886 25675 consensus_queue.cc:237] T 232ecec604ff4524af45471b49b08790 P 533e2ce30a5e48c6be272da7a20dfd63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5037e17b564c78be31bd2014807c9e" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 38889 } } peers { permanent_uuid: "533e2ce30a5e48c6be272da7a20dfd63" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46149 } } peers { permanent_uuid: "6f91fe4bd8464f3bad645b052ecd1147" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40825 } }
I20260928 06:28:28.757155 25675 tablet_replica.cc:439] T 232ecec604ff4524af45471b49b08790 P 533e2ce30a5e48c6be272da7a20dfd63: TxnStatusTablet state changed. Reason: New leader 533e2ce30a5e48c6be272da7a20dfd63. Latest consensus state: current_term: 1 leader_uuid: "533e2ce30a5e48c6be272da7a20dfd63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5037e17b564c78be31bd2014807c9e" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 38889 } } peers { permanent_uuid: "533e2ce30a5e48c6be272da7a20dfd63" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46149 } } peers { permanent_uuid: "6f91fe4bd8464f3bad645b052ecd1147" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40825 } } }
I20260928 06:28:28.757226 25675 tablet_replica.cc:442] T 232ecec604ff4524af45471b49b08790 P 533e2ce30a5e48c6be272da7a20dfd63: This TxnStatusTablet replica's current role is: LEADER
I20260928 06:28:28.757516 25690 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 06:28:28.757664 25690 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 06:28:28.757576 25162 catalog_manager.cc:5495] T 232ecec604ff4524af45471b49b08790 P 533e2ce30a5e48c6be272da7a20dfd63 reported cstate change: term changed from 0 to 1, leader changed from <none> to 533e2ce30a5e48c6be272da7a20dfd63 (127.22.225.130). New cstate: current_term: 1 leader_uuid: "533e2ce30a5e48c6be272da7a20dfd63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5037e17b564c78be31bd2014807c9e" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 38889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "533e2ce30a5e48c6be272da7a20dfd63" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f91fe4bd8464f3bad645b052ecd1147" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40825 } health_report { overall_health: UNKNOWN } } }
I20260928 06:28:28.789408 25693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:28.786655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37520 (local address 127.22.225.130:46149)
0928 06:28:28.786848 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:28.786850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:28.786857 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:28:28.786899 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:28.786900 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:28.786917 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:28.787003 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:28:28.787146 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:28.787648 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:28.788421 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:28.788728 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:28.788805 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:28.789011 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:28:28.789124 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:28:28.789229 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:28.789271 (+ 42us) server_negotiation.cc:299] Negotiation successful
0928 06:28:28.789290 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":95,"threads_started":1}
I20260928 06:28:28.789412 25685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:28.786589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.130:46149 (local address 127.0.0.1:37520)
0928 06:28:28.786669 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:28.786682 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:28.786732 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:28.787008 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:28.787009 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:28.787010 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:28:28.787062 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:28.787065 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:28.787735 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:28.787737 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:28.788317 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:28.788319 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:28.788804 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:28.788805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:28.788873 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:28.788898 (+ 25us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:28:28.789206 (+ 308us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:28:28.789215 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 06:28:28.789264 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 06:28:28.789312 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 06:28:28.791292 25319 raft_consensus.cc:1261] T 232ecec604ff4524af45471b49b08790 P 1f5037e17b564c78be31bd2014807c9e [term 1 FOLLOWER]: Refusing update from remote peer 533e2ce30a5e48c6be272da7a20dfd63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:28:28.791286 25599 raft_consensus.cc:1261] T 232ecec604ff4524af45471b49b08790 P 6f91fe4bd8464f3bad645b052ecd1147 [term 1 FOLLOWER]: Refusing update from remote peer 533e2ce30a5e48c6be272da7a20dfd63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:28:28.791651 25678 consensus_queue.cc:1035] T 232ecec604ff4524af45471b49b08790 P 533e2ce30a5e48c6be272da7a20dfd63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f91fe4bd8464f3bad645b052ecd1147" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:28:28.791841 25675 consensus_queue.cc:1035] T 232ecec604ff4524af45471b49b08790 P 533e2ce30a5e48c6be272da7a20dfd63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f5037e17b564c78be31bd2014807c9e" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:28:28.792944 25674 tablet_replica.cc:439] T 232ecec604ff4524af45471b49b08790 P 6f91fe4bd8464f3bad645b052ecd1147: TxnStatusTablet state changed. Reason: New leader 533e2ce30a5e48c6be272da7a20dfd63. Latest consensus state: current_term: 1 leader_uuid: "533e2ce30a5e48c6be272da7a20dfd63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5037e17b564c78be31bd2014807c9e" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 38889 } } peers { permanent_uuid: "533e2ce30a5e48c6be272da7a20dfd63" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46149 } } peers { permanent_uuid: "6f91fe4bd8464f3bad645b052ecd1147" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40825 } } }
I20260928 06:28:28.793041 25674 tablet_replica.cc:442] T 232ecec604ff4524af45471b49b08790 P 6f91fe4bd8464f3bad645b052ecd1147: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:28:28.793023 25676 tablet_replica.cc:439] T 232ecec604ff4524af45471b49b08790 P 1f5037e17b564c78be31bd2014807c9e: TxnStatusTablet state changed. Reason: New leader 533e2ce30a5e48c6be272da7a20dfd63. Latest consensus state: current_term: 1 leader_uuid: "533e2ce30a5e48c6be272da7a20dfd63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5037e17b564c78be31bd2014807c9e" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 38889 } } peers { permanent_uuid: "533e2ce30a5e48c6be272da7a20dfd63" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46149 } } peers { permanent_uuid: "6f91fe4bd8464f3bad645b052ecd1147" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40825 } } }
I20260928 06:28:28.793102 25676 tablet_replica.cc:442] T 232ecec604ff4524af45471b49b08790 P 1f5037e17b564c78be31bd2014807c9e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:28:28.796092 25694 mvcc.cc:205] Tried to move back new op lower bound from 7334203018406383616 to 7334203018268581888. Current Snapshot: MvccSnapshot[applied={T|T < 7334203018406383616}]
I20260928 06:28:28.796164 25678 tablet_replica.cc:439] T 232ecec604ff4524af45471b49b08790 P 533e2ce30a5e48c6be272da7a20dfd63: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "533e2ce30a5e48c6be272da7a20dfd63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5037e17b564c78be31bd2014807c9e" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 38889 } } peers { permanent_uuid: "533e2ce30a5e48c6be272da7a20dfd63" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46149 } } peers { permanent_uuid: "6f91fe4bd8464f3bad645b052ecd1147" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40825 } } }
I20260928 06:28:28.796237 25678 tablet_replica.cc:442] T 232ecec604ff4524af45471b49b08790 P 533e2ce30a5e48c6be272da7a20dfd63: This TxnStatusTablet replica's current role is: LEADER
I20260928 06:28:28.796648 25674 tablet_replica.cc:439] T 232ecec604ff4524af45471b49b08790 P 6f91fe4bd8464f3bad645b052ecd1147: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "533e2ce30a5e48c6be272da7a20dfd63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5037e17b564c78be31bd2014807c9e" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 38889 } } peers { permanent_uuid: "533e2ce30a5e48c6be272da7a20dfd63" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46149 } } peers { permanent_uuid: "6f91fe4bd8464f3bad645b052ecd1147" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40825 } } }
I20260928 06:28:28.796729 25674 tablet_replica.cc:442] T 232ecec604ff4524af45471b49b08790 P 6f91fe4bd8464f3bad645b052ecd1147: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:28:28.796947 25695 mvcc.cc:205] Tried to move back new op lower bound from 7334203018406383616 to 7334203018268581888. Current Snapshot: MvccSnapshot[applied={T|T < 7334203018406383616}]
I20260928 06:28:28.797268 25675 tablet_replica.cc:439] T 232ecec604ff4524af45471b49b08790 P 533e2ce30a5e48c6be272da7a20dfd63: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "533e2ce30a5e48c6be272da7a20dfd63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5037e17b564c78be31bd2014807c9e" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 38889 } } peers { permanent_uuid: "533e2ce30a5e48c6be272da7a20dfd63" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46149 } } peers { permanent_uuid: "6f91fe4bd8464f3bad645b052ecd1147" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40825 } } }
I20260928 06:28:28.797350 25675 tablet_replica.cc:442] T 232ecec604ff4524af45471b49b08790 P 533e2ce30a5e48c6be272da7a20dfd63: This TxnStatusTablet replica's current role is: LEADER
I20260928 06:28:28.797602 25676 tablet_replica.cc:439] T 232ecec604ff4524af45471b49b08790 P 1f5037e17b564c78be31bd2014807c9e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "533e2ce30a5e48c6be272da7a20dfd63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5037e17b564c78be31bd2014807c9e" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 38889 } } peers { permanent_uuid: "533e2ce30a5e48c6be272da7a20dfd63" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46149 } } peers { permanent_uuid: "6f91fe4bd8464f3bad645b052ecd1147" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40825 } } }
I20260928 06:28:28.797685 25676 tablet_replica.cc:442] T 232ecec604ff4524af45471b49b08790 P 1f5037e17b564c78be31bd2014807c9e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:28:28.798192 25696 mvcc.cc:205] Tried to move back new op lower bound from 7334203018406383616 to 7334203018268581888. Current Snapshot: MvccSnapshot[applied={T|T < 7334203018406383616}]
I20260928 06:28:28.804754 25693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:28.799014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37532 (local address 127.22.225.130:46149)
0928 06:28:28.799052 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:28.799054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:28.799132 (+ 78us) server_negotiation.cc:407] Connection header received
0928 06:28:28.799216 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:28.799217 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:28.799227 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:28.799308 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:28:28.799452 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:28.800001 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:28.800987 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:28.801184 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:28.801263 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:28.801407 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:28:28.801440 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:28:28.801485 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:28.804646 (+ 3161us) server_negotiation.cc:299] Negotiation successful
0928 06:28:28.804676 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 06:28:28.810794 25693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:28.807692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:54747 (local address 127.22.225.130:46149)
0928 06:28:28.807727 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:28.807729 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:28.807847 (+ 118us) server_negotiation.cc:407] Connection header received
0928 06:28:28.807896 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:28.807896 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:28.807907 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:28.807965 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:28:28.808122 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:28.808812 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:28.809917 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:28.810224 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:28.810313 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:28.810534 (+ 221us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:28:28.810566 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:28:28.810618 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:28.810674 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 06:28:28.810693 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 06:28:28.810863 25706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:28.807629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.130:46149 (local address 127.22.225.130:54747)
0928 06:28:28.807780 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:28.807790 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:28.807847 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:28.807976 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:28.807977 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:28.807979 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:28:28.808070 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:28.808071 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:28.808893 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:28.808894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:28.809811 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:28.809813 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:28.810325 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:28.810326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:28.810413 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:28.810429 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:28:28.810627 (+ 198us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:28:28.810629 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:28:28.810674 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 06:28:28.810715 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":68,"threads_started":1}
I20260928 06:28:28.813019 25319 raft_consensus.cc:1261] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 1f5037e17b564c78be31bd2014807c9e [term 1 FOLLOWER]: Refusing update from remote peer 533e2ce30a5e48c6be272da7a20dfd63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:28:28.813025 25599 raft_consensus.cc:1261] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 6f91fe4bd8464f3bad645b052ecd1147 [term 1 FOLLOWER]: Refusing update from remote peer 533e2ce30a5e48c6be272da7a20dfd63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:28:28.813210 25675 consensus_queue.cc:1035] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 533e2ce30a5e48c6be272da7a20dfd63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f91fe4bd8464f3bad645b052ecd1147" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:28:28.813320 25675 consensus_queue.cc:1035] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 533e2ce30a5e48c6be272da7a20dfd63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f5037e17b564c78be31bd2014807c9e" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:28:28.827508 25162 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:28:28.827626 25162 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:28:28.828382 25162 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9128f0ee0f504bed81b2d03646b98fd4: Sending DeleteTablet for 3 replicas of tablet 6308733bfdbd43a9a3bd5bf7ee5d33aa
I20260928 06:28:28.828701 25299 tablet_service.cc:1514] Processing DeleteTablet for tablet 6308733bfdbd43a9a3bd5bf7ee5d33aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:28:28 UTC) from {username='kudu', principal='kudu/127.22.225.190@KRBTEST.COM'} at 127.0.0.1:56874
I20260928 06:28:28.828737 25567 tablet_service.cc:1514] Processing DeleteTablet for tablet 6308733bfdbd43a9a3bd5bf7ee5d33aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:28:28 UTC) from {username='kudu', principal='kudu/127.22.225.190@KRBTEST.COM'} at 127.0.0.1:43402
I20260928 06:28:28.828930 25713 tablet_replica.cc:330] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 6f91fe4bd8464f3bad645b052ecd1147: stopping tablet replica
I20260928 06:28:28.829018 25713 raft_consensus.cc:2229] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 6f91fe4bd8464f3bad645b052ecd1147 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:28:28.829079 25713 raft_consensus.cc:2258] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 6f91fe4bd8464f3bad645b052ecd1147 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:28:28.829013 25162 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43804:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:28:28.829139 25162 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:28:28.829231 25712 tablet_replica.cc:330] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 1f5037e17b564c78be31bd2014807c9e: stopping tablet replica
I20260928 06:28:28.829320 25712 raft_consensus.cc:2229] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 1f5037e17b564c78be31bd2014807c9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:28:28.829371 25712 raft_consensus.cc:2258] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 1f5037e17b564c78be31bd2014807c9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:28:28.829430 25439 tablet_service.cc:1514] Processing DeleteTablet for tablet 6308733bfdbd43a9a3bd5bf7ee5d33aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:28:28 UTC) from {username='kudu', principal='kudu/127.22.225.190@KRBTEST.COM'} at 127.0.0.1:37518
I20260928 06:28:28.829653 25714 tablet_replica.cc:330] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 533e2ce30a5e48c6be272da7a20dfd63: stopping tablet replica
I20260928 06:28:28.829744 25714 raft_consensus.cc:2229] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 533e2ce30a5e48c6be272da7a20dfd63 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:28:28.829824 25714 raft_consensus.cc:2258] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 533e2ce30a5e48c6be272da7a20dfd63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:28:28.829878 25713 ts_tablet_manager.cc:1905] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 6f91fe4bd8464f3bad645b052ecd1147: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:28:28.830181 25714 ts_tablet_manager.cc:1905] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 533e2ce30a5e48c6be272da7a20dfd63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:28:28.830410 25567 tablet_service.cc:1467] Processing CreateTablet for tablet dbeddc7f4bc9497a93b720acaef6b558 (DEFAULT_TABLE table=test-table [id=5a7150c819cd40e79356e7f6df3e5cbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:28:28.830505 25567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbeddc7f4bc9497a93b720acaef6b558. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:28.831699 25713 ts_tablet_manager.cc:1918] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 6f91fe4bd8464f3bad645b052ecd1147: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:28:28.831717 25668 tablet_bootstrap.cc:492] T dbeddc7f4bc9497a93b720acaef6b558 P 6f91fe4bd8464f3bad645b052ecd1147: Bootstrap starting.
I20260928 06:28:28.831765 25713 log.cc:1196] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 6f91fe4bd8464f3bad645b052ecd1147: Deleting WAL directory at /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/ts-2/wal/wals/6308733bfdbd43a9a3bd5bf7ee5d33aa
I20260928 06:28:28.831772 25714 ts_tablet_manager.cc:1918] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 533e2ce30a5e48c6be272da7a20dfd63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:28:28.831828 25714 log.cc:1196] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 533e2ce30a5e48c6be272da7a20dfd63: Deleting WAL directory at /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/ts-1/wal/wals/6308733bfdbd43a9a3bd5bf7ee5d33aa
I20260928 06:28:28.832023 25713 ts_tablet_manager.cc:1939] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 6f91fe4bd8464f3bad645b052ecd1147: Deleting consensus metadata
I20260928 06:28:28.832118 25714 ts_tablet_manager.cc:1939] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 533e2ce30a5e48c6be272da7a20dfd63: Deleting consensus metadata
I20260928 06:28:28.832140 25668 tablet_bootstrap.cc:654] T dbeddc7f4bc9497a93b720acaef6b558 P 6f91fe4bd8464f3bad645b052ecd1147: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:28.832578 25146 catalog_manager.cc:4841] TS 6f91fe4bd8464f3bad645b052ecd1147 (127.22.225.131:40825): tablet 6308733bfdbd43a9a3bd5bf7ee5d33aa (table test-table [id=be48279697c247b996057cdac7559108]) successfully deleted
I20260928 06:28:28.832669 25712 ts_tablet_manager.cc:1905] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 1f5037e17b564c78be31bd2014807c9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:28:28.832733 25299 tablet_service.cc:1467] Processing CreateTablet for tablet dbeddc7f4bc9497a93b720acaef6b558 (DEFAULT_TABLE table=test-table [id=5a7150c819cd40e79356e7f6df3e5cbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:28:28.832837 25299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbeddc7f4bc9497a93b720acaef6b558. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:28.834093 25670 tablet_bootstrap.cc:492] T dbeddc7f4bc9497a93b720acaef6b558 P 1f5037e17b564c78be31bd2014807c9e: Bootstrap starting.
I20260928 06:28:28.834327 25712 ts_tablet_manager.cc:1918] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 1f5037e17b564c78be31bd2014807c9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:28:28.834391 25712 log.cc:1196] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 1f5037e17b564c78be31bd2014807c9e: Deleting WAL directory at /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/ts-0/wal/wals/6308733bfdbd43a9a3bd5bf7ee5d33aa
I20260928 06:28:28.834443 25670 tablet_bootstrap.cc:654] T dbeddc7f4bc9497a93b720acaef6b558 P 1f5037e17b564c78be31bd2014807c9e: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:28.834653 25712 ts_tablet_manager.cc:1939] T 6308733bfdbd43a9a3bd5bf7ee5d33aa P 1f5037e17b564c78be31bd2014807c9e: Deleting consensus metadata
I20260928 06:28:28.835182 25146 catalog_manager.cc:4841] TS 1f5037e17b564c78be31bd2014807c9e (127.22.225.129:38889): tablet 6308733bfdbd43a9a3bd5bf7ee5d33aa (table test-table [id=be48279697c247b996057cdac7559108]) successfully deleted
I20260928 06:28:28.835395 25147 catalog_manager.cc:4841] TS 533e2ce30a5e48c6be272da7a20dfd63 (127.22.225.130:46149): tablet 6308733bfdbd43a9a3bd5bf7ee5d33aa (table test-table [id=be48279697c247b996057cdac7559108]) successfully deleted
I20260928 06:28:28.835485 25439 tablet_service.cc:1467] Processing CreateTablet for tablet dbeddc7f4bc9497a93b720acaef6b558 (DEFAULT_TABLE table=test-table [id=5a7150c819cd40e79356e7f6df3e5cbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:28:28.835578 25439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbeddc7f4bc9497a93b720acaef6b558. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:28.835652 25670 tablet_bootstrap.cc:492] T dbeddc7f4bc9497a93b720acaef6b558 P 1f5037e17b564c78be31bd2014807c9e: No bootstrap required, opened a new log
I20260928 06:28:28.835690 25670 ts_tablet_manager.cc:1397] T dbeddc7f4bc9497a93b720acaef6b558 P 1f5037e17b564c78be31bd2014807c9e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 06:28:28.835821 25670 raft_consensus.cc:348] T dbeddc7f4bc9497a93b720acaef6b558 P 1f5037e17b564c78be31bd2014807c9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5037e17b564c78be31bd2014807c9e" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 38889 } } peers { permanent_uuid: "533e2ce30a5e48c6be272da7a20dfd63" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46149 } } peers { permanent_uuid: "6f91fe4bd8464f3bad645b052ecd1147" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40825 } }
I20260928 06:28:28.835892 25670 raft_consensus.cc:374] T dbeddc7f4bc9497a93b720acaef6b558 P 1f5037e17b564c78be31bd2014807c9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:28.835911 25670 raft_consensus.cc:729] T dbeddc7f4bc9497a93b720acaef6b558 P 1f5037e17b564c78be31bd2014807c9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f5037e17b564c78be31bd2014807c9e, State: Initialized, Role: FOLLOWER
I20260928 06:28:28.835955 25670 consensus_queue.cc:260] T dbeddc7f4bc9497a93b720acaef6b558 P 1f5037e17b564c78be31bd2014807c9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5037e17b564c78be31bd2014807c9e" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 38889 } } peers { permanent_uuid: "533e2ce30a5e48c6be272da7a20dfd63" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46149 } } peers { permanent_uuid: "6f91fe4bd8464f3bad645b052ecd1147" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40825 } }
I20260928 06:28:28.836036 25670 ts_tablet_manager.cc:1428] T dbeddc7f4bc9497a93b720acaef6b558 P 1f5037e17b564c78be31bd2014807c9e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:28.836815 25669 tablet_bootstrap.cc:492] T dbeddc7f4bc9497a93b720acaef6b558 P 533e2ce30a5e48c6be272da7a20dfd63: Bootstrap starting.
I20260928 06:28:28.836944 25668 tablet_bootstrap.cc:492] T dbeddc7f4bc9497a93b720acaef6b558 P 6f91fe4bd8464f3bad645b052ecd1147: No bootstrap required, opened a new log
I20260928 06:28:28.836993 25668 ts_tablet_manager.cc:1397] T dbeddc7f4bc9497a93b720acaef6b558 P 6f91fe4bd8464f3bad645b052ecd1147: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260928 06:28:28.837134 25668 raft_consensus.cc:348] T dbeddc7f4bc9497a93b720acaef6b558 P 6f91fe4bd8464f3bad645b052ecd1147 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5037e17b564c78be31bd2014807c9e" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 38889 } } peers { permanent_uuid: "533e2ce30a5e48c6be272da7a20dfd63" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46149 } } peers { permanent_uuid: "6f91fe4bd8464f3bad645b052ecd1147" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40825 } }
I20260928 06:28:28.837188 25669 tablet_bootstrap.cc:654] T dbeddc7f4bc9497a93b720acaef6b558 P 533e2ce30a5e48c6be272da7a20dfd63: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:28.837196 25668 raft_consensus.cc:374] T dbeddc7f4bc9497a93b720acaef6b558 P 6f91fe4bd8464f3bad645b052ecd1147 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:28.837216 25668 raft_consensus.cc:729] T dbeddc7f4bc9497a93b720acaef6b558 P 6f91fe4bd8464f3bad645b052ecd1147 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f91fe4bd8464f3bad645b052ecd1147, State: Initialized, Role: FOLLOWER
I20260928 06:28:28.837257 25668 consensus_queue.cc:260] T dbeddc7f4bc9497a93b720acaef6b558 P 6f91fe4bd8464f3bad645b052ecd1147 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5037e17b564c78be31bd2014807c9e" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 38889 } } peers { permanent_uuid: "533e2ce30a5e48c6be272da7a20dfd63" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46149 } } peers { permanent_uuid: "6f91fe4bd8464f3bad645b052ecd1147" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40825 } }
I20260928 06:28:28.837338 25668 ts_tablet_manager.cc:1428] T dbeddc7f4bc9497a93b720acaef6b558 P 6f91fe4bd8464f3bad645b052ecd1147: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:28.841074 25669 tablet_bootstrap.cc:492] T dbeddc7f4bc9497a93b720acaef6b558 P 533e2ce30a5e48c6be272da7a20dfd63: No bootstrap required, opened a new log
I20260928 06:28:28.841136 25669 ts_tablet_manager.cc:1397] T dbeddc7f4bc9497a93b720acaef6b558 P 533e2ce30a5e48c6be272da7a20dfd63: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260928 06:28:28.841289 25669 raft_consensus.cc:348] T dbeddc7f4bc9497a93b720acaef6b558 P 533e2ce30a5e48c6be272da7a20dfd63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5037e17b564c78be31bd2014807c9e" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 38889 } } peers { permanent_uuid: "533e2ce30a5e48c6be272da7a20dfd63" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46149 } } peers { permanent_uuid: "6f91fe4bd8464f3bad645b052ecd1147" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40825 } }
I20260928 06:28:28.841364 25669 raft_consensus.cc:374] T dbeddc7f4bc9497a93b720acaef6b558 P 533e2ce30a5e48c6be272da7a20dfd63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:28.841398 25669 raft_consensus.cc:729] T dbeddc7f4bc9497a93b720acaef6b558 P 533e2ce30a5e48c6be272da7a20dfd63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 533e2ce30a5e48c6be272da7a20dfd63, State: Initialized, Role: FOLLOWER
I20260928 06:28:28.841445 25669 consensus_queue.cc:260] T dbeddc7f4bc9497a93b720acaef6b558 P 533e2ce30a5e48c6be272da7a20dfd63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5037e17b564c78be31bd2014807c9e" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 38889 } } peers { permanent_uuid: "533e2ce30a5e48c6be272da7a20dfd63" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46149 } } peers { permanent_uuid: "6f91fe4bd8464f3bad645b052ecd1147" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40825 } }
I20260928 06:28:28.841547 25669 ts_tablet_manager.cc:1428] T dbeddc7f4bc9497a93b720acaef6b558 P 533e2ce30a5e48c6be272da7a20dfd63: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 06:28:28.857131 25676 raft_consensus.cc:482] T dbeddc7f4bc9497a93b720acaef6b558 P 1f5037e17b564c78be31bd2014807c9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:28:28.857234 25676 raft_consensus.cc:504] T dbeddc7f4bc9497a93b720acaef6b558 P 1f5037e17b564c78be31bd2014807c9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5037e17b564c78be31bd2014807c9e" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 38889 } } peers { permanent_uuid: "533e2ce30a5e48c6be272da7a20dfd63" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46149 } } peers { permanent_uuid: "6f91fe4bd8464f3bad645b052ecd1147" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40825 } }
I20260928 06:28:28.857590 25676 leader_election.cc:288] T dbeddc7f4bc9497a93b720acaef6b558 P 1f5037e17b564c78be31bd2014807c9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 533e2ce30a5e48c6be272da7a20dfd63 (127.22.225.130:46149), 6f91fe4bd8464f3bad645b052ecd1147 (127.22.225.131:40825)
I20260928 06:28:28.861517 25667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:28.857823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:43705 (local address 127.22.225.131:40825)
0928 06:28:28.857869 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:28.857871 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:28.858040 (+ 169us) server_negotiation.cc:407] Connection header received
0928 06:28:28.858092 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:28.858093 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:28.858103 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:28.858166 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:28:28.858520 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:28.859233 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:28.860631 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:28.861131 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:28.861182 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:28.861192 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:28.861406 (+ 214us) server_negotiation.cc:299] Negotiation successful
0928 06:28:28.861449 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:28:28.861655 25693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:28.857908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:42419 (local address 127.22.225.130:46149)
0928 06:28:28.857960 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:28.857962 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:28.858159 (+ 197us) server_negotiation.cc:407] Connection header received
0928 06:28:28.858242 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:28.858242 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:28.858252 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:28.858278 (+ 26us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:28:28.858637 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:28.859290 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:28.860639 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:28.861237 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:28.861299 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:28.861307 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:28.861548 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 06:28:28.861586 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 06:28:28.861769 25718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:28.857738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.131:40825 (local address 127.22.225.129:43705)
0928 06:28:28.857955 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:28.857968 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:28.858030 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:28.858348 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:28.858349 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:28.858352 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:28:28.858425 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:28.858427 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:28.859331 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:28.859331 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:28.860489 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:28.860492 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:28.861253 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:28.861253 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:28.861323 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:28.861326 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:28:28.861644 (+ 318us) client_negotiation.cc:240] Negotiation successful
0928 06:28:28.861688 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":84,"threads_started":1}
I20260928 06:28:28.862111 25599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbeddc7f4bc9497a93b720acaef6b558" candidate_uuid: "1f5037e17b564c78be31bd2014807c9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f91fe4bd8464f3bad645b052ecd1147" is_pre_election: true
I20260928 06:28:28.862211 25599 raft_consensus.cc:2454] T dbeddc7f4bc9497a93b720acaef6b558 P 6f91fe4bd8464f3bad645b052ecd1147 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f5037e17b564c78be31bd2014807c9e in term 0.
I20260928 06:28:28.862341 25719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:28.857738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.130:46149 (local address 127.22.225.129:42419)
0928 06:28:28.858117 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:28.858124 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:28.858177 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:28.858489 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:28.858489 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:28.858491 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:28:28.858561 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:28.858562 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:28.859416 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:28.859417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:28.860514 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:28.860516 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:28.861388 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:28.861389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:28.861470 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:28.861473 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:28:28.862196 (+ 723us) client_negotiation.cc:240] Negotiation successful
0928 06:28:28.862244 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"mutex_wait_us":15,"thread_start_us":72,"threads_started":1}
I20260928 06:28:28.862579 25459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbeddc7f4bc9497a93b720acaef6b558" candidate_uuid: "1f5037e17b564c78be31bd2014807c9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "533e2ce30a5e48c6be272da7a20dfd63" is_pre_election: true
I20260928 06:28:28.862612 25250 leader_election.cc:302] T dbeddc7f4bc9497a93b720acaef6b558 P 1f5037e17b564c78be31bd2014807c9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f5037e17b564c78be31bd2014807c9e, 6f91fe4bd8464f3bad645b052ecd1147; no voters:
I20260928 06:28:28.862704 25459 raft_consensus.cc:2454] T dbeddc7f4bc9497a93b720acaef6b558 P 533e2ce30a5e48c6be272da7a20dfd63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f5037e17b564c78be31bd2014807c9e in term 0.
I20260928 06:28:28.862735 25676 raft_consensus.cc:2781] T dbeddc7f4bc9497a93b720acaef6b558 P 1f5037e17b564c78be31bd2014807c9e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:28:28.862766 25676 raft_consensus.cc:482] T dbeddc7f4bc9497a93b720acaef6b558 P 1f5037e17b564c78be31bd2014807c9e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:28:28.862785 25676 raft_consensus.cc:3037] T dbeddc7f4bc9497a93b720acaef6b558 P 1f5037e17b564c78be31bd2014807c9e [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:28.863622 25676 raft_consensus.cc:504] T dbeddc7f4bc9497a93b720acaef6b558 P 1f5037e17b564c78be31bd2014807c9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5037e17b564c78be31bd2014807c9e" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 38889 } } peers { permanent_uuid: "533e2ce30a5e48c6be272da7a20dfd63" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46149 } } peers { permanent_uuid: "6f91fe4bd8464f3bad645b052ecd1147" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40825 } }
I20260928 06:28:28.863755 25676 leader_election.cc:288] T dbeddc7f4bc9497a93b720acaef6b558 P 1f5037e17b564c78be31bd2014807c9e [CANDIDATE]: Term 1 election: Requested vote from peers 533e2ce30a5e48c6be272da7a20dfd63 (127.22.225.130:46149), 6f91fe4bd8464f3bad645b052ecd1147 (127.22.225.131:40825)
I20260928 06:28:28.863909 25459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbeddc7f4bc9497a93b720acaef6b558" candidate_uuid: "1f5037e17b564c78be31bd2014807c9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "533e2ce30a5e48c6be272da7a20dfd63"
I20260928 06:28:28.863982 25459 raft_consensus.cc:3037] T dbeddc7f4bc9497a93b720acaef6b558 P 533e2ce30a5e48c6be272da7a20dfd63 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:28.864090 25599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbeddc7f4bc9497a93b720acaef6b558" candidate_uuid: "1f5037e17b564c78be31bd2014807c9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f91fe4bd8464f3bad645b052ecd1147"
I20260928 06:28:28.864166 25599 raft_consensus.cc:3037] T dbeddc7f4bc9497a93b720acaef6b558 P 6f91fe4bd8464f3bad645b052ecd1147 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:28.864840 25459 raft_consensus.cc:2454] T dbeddc7f4bc9497a93b720acaef6b558 P 533e2ce30a5e48c6be272da7a20dfd63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f5037e17b564c78be31bd2014807c9e in term 1.
I20260928 06:28:28.864926 25599 raft_consensus.cc:2454] T dbeddc7f4bc9497a93b720acaef6b558 P 6f91fe4bd8464f3bad645b052ecd1147 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f5037e17b564c78be31bd2014807c9e in term 1.
I20260928 06:28:28.865104 25250 leader_election.cc:302] T dbeddc7f4bc9497a93b720acaef6b558 P 1f5037e17b564c78be31bd2014807c9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f5037e17b564c78be31bd2014807c9e, 6f91fe4bd8464f3bad645b052ecd1147; no voters:
I20260928 06:28:28.865267 25676 raft_consensus.cc:2781] T dbeddc7f4bc9497a93b720acaef6b558 P 1f5037e17b564c78be31bd2014807c9e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:28:28.865432 25676 raft_consensus.cc:686] T dbeddc7f4bc9497a93b720acaef6b558 P 1f5037e17b564c78be31bd2014807c9e [term 1 LEADER]: Becoming Leader. State: Replica: 1f5037e17b564c78be31bd2014807c9e, State: Running, Role: LEADER
I20260928 06:28:28.865512 25676 consensus_queue.cc:237] T dbeddc7f4bc9497a93b720acaef6b558 P 1f5037e17b564c78be31bd2014807c9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5037e17b564c78be31bd2014807c9e" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 38889 } } peers { permanent_uuid: "533e2ce30a5e48c6be272da7a20dfd63" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46149 } } peers { permanent_uuid: "6f91fe4bd8464f3bad645b052ecd1147" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40825 } }
I20260928 06:28:28.866201 25161 catalog_manager.cc:5495] T dbeddc7f4bc9497a93b720acaef6b558 P 1f5037e17b564c78be31bd2014807c9e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f5037e17b564c78be31bd2014807c9e (127.22.225.129). New cstate: current_term: 1 leader_uuid: "1f5037e17b564c78be31bd2014807c9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5037e17b564c78be31bd2014807c9e" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 38889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "533e2ce30a5e48c6be272da7a20dfd63" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f91fe4bd8464f3bad645b052ecd1147" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40825 } health_report { overall_health: UNKNOWN } } }
I20260928 06:28:28.879809 25664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:28.876678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56884 (local address 127.22.225.129:38889)
0928 06:28:28.876724 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:28.876726 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:28.876736 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:28:28.876793 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:28.876794 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:28.876816 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:28.876882 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:28:28.877055 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:28.877808 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:28.878867 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:28.879153 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:28.879225 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:28.879407 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:28:28.879545 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:28:28.879618 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:28.879710 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 06:28:28.879732 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:28:28.885614 25693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:28.882629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:60745 (local address 127.22.225.130:46149)
0928 06:28:28.882681 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:28.882683 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:28.882691 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:28:28.882783 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:28.882784 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:28.882795 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:28.882856 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:28:28.883004 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:28.883651 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:28.884778 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:28.885066 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:28.885142 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:28.885349 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:28:28.885386 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:28:28.885443 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:28.885515 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 06:28:28.885534 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 06:28:28.886060 25724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:28.882483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.130:46149 (local address 127.22.225.129:60745)
0928 06:28:28.882630 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:28.882642 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:28.882705 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:28.882872 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:28.882873 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:28.882875 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:28:28.882938 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:28.882939 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:28.883742 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:28.883743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:28.884692 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:28.884694 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:28.885161 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:28.885162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:28.885236 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:28.885251 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:28:28.885457 (+ 206us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:28:28.885459 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:28:28.885500 (+ 41us) client_negotiation.cc:240] Negotiation successful
0928 06:28:28.885542 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":80,"threads_started":1}
I20260928 06:28:28.888490 25599 raft_consensus.cc:1261] T dbeddc7f4bc9497a93b720acaef6b558 P 6f91fe4bd8464f3bad645b052ecd1147 [term 1 FOLLOWER]: Refusing update from remote peer 1f5037e17b564c78be31bd2014807c9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:28:28.888558 25459 raft_consensus.cc:1261] T dbeddc7f4bc9497a93b720acaef6b558 P 533e2ce30a5e48c6be272da7a20dfd63 [term 1 FOLLOWER]: Refusing update from remote peer 1f5037e17b564c78be31bd2014807c9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:28:28.888782 25720 consensus_queue.cc:1035] T dbeddc7f4bc9497a93b720acaef6b558 P 1f5037e17b564c78be31bd2014807c9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f91fe4bd8464f3bad645b052ecd1147" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:28:28.888937 25720 consensus_queue.cc:1035] T dbeddc7f4bc9497a93b720acaef6b558 P 1f5037e17b564c78be31bd2014807c9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "533e2ce30a5e48c6be272da7a20dfd63" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:28:28.897749 25706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:28.894585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.129:38889 (local address 127.22.225.130:56605)
0928 06:28:28.894633 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:28.894644 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:28.894681 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:28.895123 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:28.895124 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:28.895125 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:28:28.895176 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:28.895177 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:28.895999 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:28.896000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:28.896815 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:28.896818 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:28.897311 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:28.897312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:28.897373 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:28.897380 (+ 7us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:28:28.897589 (+ 209us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:28:28.897592 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:28:28.897631 (+ 39us) client_negotiation.cc:240] Negotiation successful
0928 06:28:28.897677 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 06:28:28.898048 25664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:28.894889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:56605 (local address 127.22.225.129:38889)
0928 06:28:28.894925 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:28.894926 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:28.894933 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:28:28.894953 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:28.894954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:28.894964 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:28.895019 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:28:28.895259 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:28.895921 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:28.896934 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:28.897226 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:28.897297 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:28.897462 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:28:28.897494 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:28:28.897563 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:28.897962 (+ 399us) server_negotiation.cc:299] Negotiation successful
0928 06:28:28.897983 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 06:28:28.898432 25696 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 06:28:30.426172 25161 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:28:30.426282 25161 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:28:30.427115 25161 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9128f0ee0f504bed81b2d03646b98fd4: Sending DeleteTablet for 3 replicas of tablet dbeddc7f4bc9497a93b720acaef6b558
I20260928 06:28:30.427423 25439 tablet_service.cc:1514] Processing DeleteTablet for tablet dbeddc7f4bc9497a93b720acaef6b558 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:28:30 UTC) from {username='kudu', principal='kudu/127.22.225.190@KRBTEST.COM'} at 127.0.0.1:37518
I20260928 06:28:30.427811 25750 tablet_replica.cc:330] T dbeddc7f4bc9497a93b720acaef6b558 P 533e2ce30a5e48c6be272da7a20dfd63: stopping tablet replica
I20260928 06:28:30.427942 25750 raft_consensus.cc:2229] T dbeddc7f4bc9497a93b720acaef6b558 P 533e2ce30a5e48c6be272da7a20dfd63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:28:30.428043 25750 raft_consensus.cc:2258] T dbeddc7f4bc9497a93b720acaef6b558 P 533e2ce30a5e48c6be272da7a20dfd63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:28:30.428397 25750 ts_tablet_manager.cc:1905] T dbeddc7f4bc9497a93b720acaef6b558 P 533e2ce30a5e48c6be272da7a20dfd63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:28:30.428416 25299 tablet_service.cc:1514] Processing DeleteTablet for tablet dbeddc7f4bc9497a93b720acaef6b558 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:28:30 UTC) from {username='kudu', principal='kudu/127.22.225.190@KRBTEST.COM'} at 127.0.0.1:56874
I20260928 06:28:30.428453 25567 tablet_service.cc:1514] Processing DeleteTablet for tablet dbeddc7f4bc9497a93b720acaef6b558 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:28:30 UTC) from {username='kudu', principal='kudu/127.22.225.190@KRBTEST.COM'} at 127.0.0.1:43402
I20260928 06:28:30.428618 25751 tablet_replica.cc:330] T dbeddc7f4bc9497a93b720acaef6b558 P 1f5037e17b564c78be31bd2014807c9e: stopping tablet replica
I20260928 06:28:30.428681 25751 raft_consensus.cc:2229] T dbeddc7f4bc9497a93b720acaef6b558 P 1f5037e17b564c78be31bd2014807c9e [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:28:30.428784 25751 raft_consensus.cc:2258] T dbeddc7f4bc9497a93b720acaef6b558 P 1f5037e17b564c78be31bd2014807c9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:28:30.429003 25751 ts_tablet_manager.cc:1905] T dbeddc7f4bc9497a93b720acaef6b558 P 1f5037e17b564c78be31bd2014807c9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:28:30.430488 25751 ts_tablet_manager.cc:1918] T dbeddc7f4bc9497a93b720acaef6b558 P 1f5037e17b564c78be31bd2014807c9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 06:28:30.430488 25750 ts_tablet_manager.cc:1918] T dbeddc7f4bc9497a93b720acaef6b558 P 533e2ce30a5e48c6be272da7a20dfd63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 06:28:30.430553 25750 log.cc:1196] T dbeddc7f4bc9497a93b720acaef6b558 P 533e2ce30a5e48c6be272da7a20dfd63: Deleting WAL directory at /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/ts-1/wal/wals/dbeddc7f4bc9497a93b720acaef6b558
I20260928 06:28:30.430553 25751 log.cc:1196] T dbeddc7f4bc9497a93b720acaef6b558 P 1f5037e17b564c78be31bd2014807c9e: Deleting WAL directory at /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/ts-0/wal/wals/dbeddc7f4bc9497a93b720acaef6b558
I20260928 06:28:30.430783 25751 ts_tablet_manager.cc:1939] T dbeddc7f4bc9497a93b720acaef6b558 P 1f5037e17b564c78be31bd2014807c9e: Deleting consensus metadata
I20260928 06:28:30.430783 25750 ts_tablet_manager.cc:1939] T dbeddc7f4bc9497a93b720acaef6b558 P 533e2ce30a5e48c6be272da7a20dfd63: Deleting consensus metadata
I20260928 06:28:30.431274 25146 catalog_manager.cc:4841] TS 1f5037e17b564c78be31bd2014807c9e (127.22.225.129:38889): tablet dbeddc7f4bc9497a93b720acaef6b558 (table test-table [id=5a7150c819cd40e79356e7f6df3e5cbc]) successfully deleted
I20260928 06:28:30.431373 25147 catalog_manager.cc:4841] TS 533e2ce30a5e48c6be272da7a20dfd63 (127.22.225.130:46149): tablet dbeddc7f4bc9497a93b720acaef6b558 (table test-table [id=5a7150c819cd40e79356e7f6df3e5cbc]) successfully deleted
I20260928 06:28:30.432117 25752 tablet_replica.cc:330] T dbeddc7f4bc9497a93b720acaef6b558 P 6f91fe4bd8464f3bad645b052ecd1147: stopping tablet replica
I20260928 06:28:30.432200 25752 raft_consensus.cc:2229] T dbeddc7f4bc9497a93b720acaef6b558 P 6f91fe4bd8464f3bad645b052ecd1147 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:28:30.432247 25752 raft_consensus.cc:2258] T dbeddc7f4bc9497a93b720acaef6b558 P 6f91fe4bd8464f3bad645b052ecd1147 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:28:30.432493 25752 ts_tablet_manager.cc:1905] T dbeddc7f4bc9497a93b720acaef6b558 P 6f91fe4bd8464f3bad645b052ecd1147: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:28:30.434671 25752 ts_tablet_manager.cc:1918] T dbeddc7f4bc9497a93b720acaef6b558 P 6f91fe4bd8464f3bad645b052ecd1147: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 06:28:30.434732 25752 log.cc:1196] T dbeddc7f4bc9497a93b720acaef6b558 P 6f91fe4bd8464f3bad645b052ecd1147: Deleting WAL directory at /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/ts-2/wal/wals/dbeddc7f4bc9497a93b720acaef6b558
I20260928 06:28:30.434950 25752 ts_tablet_manager.cc:1939] T dbeddc7f4bc9497a93b720acaef6b558 P 6f91fe4bd8464f3bad645b052ecd1147: Deleting consensus metadata
I20260928 06:28:30.435405 25146 catalog_manager.cc:4841] TS 6f91fe4bd8464f3bad645b052ecd1147 (127.22.225.131:40825): tablet dbeddc7f4bc9497a93b720acaef6b558 (table test-table [id=5a7150c819cd40e79356e7f6df3e5cbc]) successfully deleted
Sep 28 06:28:30 dist-test-slave-d59m krb5kdc[25116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576910, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 06:28:30 dist-test-slave-d59m krb5kdc[25116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576910, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:28:30.456393 25760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:30.448687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43816 (local address 127.22.225.190:36347)
0928 06:28:30.448869 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:30.448872 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:30.448883 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:28:30.448920 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:30.448920 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:30.448934 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:30.449010 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:30.449552 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:30.450251 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:30.450959 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:30.451229 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:30.451296 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:30.453731 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:30.453732 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:30.453733 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:30.453740 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:30.455343 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:30.455664 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:30.455665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:30.455665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:30.455727 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:30.455923 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:30.455923 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:30.455924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:30.456048 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:30.456143 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:30.456226 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 06:28:30.456301 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":102,"threads_started":1}
I20260928 06:28:30.458052 25161 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43816:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:28:30.458228 25161 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:28:30.459576 25439 tablet_service.cc:1467] Processing CreateTablet for tablet c7b68fa203b14738a791526c243eb782 (DEFAULT_TABLE table=test-table [id=37da873defb541b88650bfcb4ce5804c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:28:30.459584 25299 tablet_service.cc:1467] Processing CreateTablet for tablet c7b68fa203b14738a791526c243eb782 (DEFAULT_TABLE table=test-table [id=37da873defb541b88650bfcb4ce5804c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:28:30.459589 25567 tablet_service.cc:1467] Processing CreateTablet for tablet c7b68fa203b14738a791526c243eb782 (DEFAULT_TABLE table=test-table [id=37da873defb541b88650bfcb4ce5804c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:28:30.459714 25567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7b68fa203b14738a791526c243eb782. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:30.459713 25299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7b68fa203b14738a791526c243eb782. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:30.459713 25439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7b68fa203b14738a791526c243eb782. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:30.460759 25761 tablet_bootstrap.cc:492] T c7b68fa203b14738a791526c243eb782 P 6f91fe4bd8464f3bad645b052ecd1147: Bootstrap starting.
I20260928 06:28:30.461102 25761 tablet_bootstrap.cc:654] T c7b68fa203b14738a791526c243eb782 P 6f91fe4bd8464f3bad645b052ecd1147: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:30.461427 25763 tablet_bootstrap.cc:492] T c7b68fa203b14738a791526c243eb782 P 1f5037e17b564c78be31bd2014807c9e: Bootstrap starting.
I20260928 06:28:30.461776 25761 tablet_bootstrap.cc:492] T c7b68fa203b14738a791526c243eb782 P 6f91fe4bd8464f3bad645b052ecd1147: No bootstrap required, opened a new log
I20260928 06:28:30.461828 25761 ts_tablet_manager.cc:1397] T c7b68fa203b14738a791526c243eb782 P 6f91fe4bd8464f3bad645b052ecd1147: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:28:30.461953 25763 tablet_bootstrap.cc:654] T c7b68fa203b14738a791526c243eb782 P 1f5037e17b564c78be31bd2014807c9e: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:30.462119 25761 raft_consensus.cc:348] T c7b68fa203b14738a791526c243eb782 P 6f91fe4bd8464f3bad645b052ecd1147 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5037e17b564c78be31bd2014807c9e" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 38889 } } peers { permanent_uuid: "533e2ce30a5e48c6be272da7a20dfd63" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46149 } } peers { permanent_uuid: "6f91fe4bd8464f3bad645b052ecd1147" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40825 } }
I20260928 06:28:30.462203 25761 raft_consensus.cc:374] T c7b68fa203b14738a791526c243eb782 P 6f91fe4bd8464f3bad645b052ecd1147 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:30.462221 25761 raft_consensus.cc:729] T c7b68fa203b14738a791526c243eb782 P 6f91fe4bd8464f3bad645b052ecd1147 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f91fe4bd8464f3bad645b052ecd1147, State: Initialized, Role: FOLLOWER
I20260928 06:28:30.462275 25761 consensus_queue.cc:260] T c7b68fa203b14738a791526c243eb782 P 6f91fe4bd8464f3bad645b052ecd1147 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5037e17b564c78be31bd2014807c9e" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 38889 } } peers { permanent_uuid: "533e2ce30a5e48c6be272da7a20dfd63" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46149 } } peers { permanent_uuid: "6f91fe4bd8464f3bad645b052ecd1147" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40825 } }
I20260928 06:28:30.462281 25762 tablet_bootstrap.cc:492] T c7b68fa203b14738a791526c243eb782 P 533e2ce30a5e48c6be272da7a20dfd63: Bootstrap starting.
I20260928 06:28:30.462433 25761 ts_tablet_manager.cc:1428] T c7b68fa203b14738a791526c243eb782 P 6f91fe4bd8464f3bad645b052ecd1147: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:28:30.462827 25763 tablet_bootstrap.cc:492] T c7b68fa203b14738a791526c243eb782 P 1f5037e17b564c78be31bd2014807c9e: No bootstrap required, opened a new log
I20260928 06:28:30.462875 25763 ts_tablet_manager.cc:1397] T c7b68fa203b14738a791526c243eb782 P 1f5037e17b564c78be31bd2014807c9e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:28:30.462986 25763 raft_consensus.cc:348] T c7b68fa203b14738a791526c243eb782 P 1f5037e17b564c78be31bd2014807c9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5037e17b564c78be31bd2014807c9e" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 38889 } } peers { permanent_uuid: "533e2ce30a5e48c6be272da7a20dfd63" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46149 } } peers { permanent_uuid: "6f91fe4bd8464f3bad645b052ecd1147" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40825 } }
I20260928 06:28:30.463057 25763 raft_consensus.cc:374] T c7b68fa203b14738a791526c243eb782 P 1f5037e17b564c78be31bd2014807c9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:30.463078 25763 raft_consensus.cc:729] T c7b68fa203b14738a791526c243eb782 P 1f5037e17b564c78be31bd2014807c9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f5037e17b564c78be31bd2014807c9e, State: Initialized, Role: FOLLOWER
I20260928 06:28:30.463140 25763 consensus_queue.cc:260] T c7b68fa203b14738a791526c243eb782 P 1f5037e17b564c78be31bd2014807c9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5037e17b564c78be31bd2014807c9e" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 38889 } } peers { permanent_uuid: "533e2ce30a5e48c6be272da7a20dfd63" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46149 } } peers { permanent_uuid: "6f91fe4bd8464f3bad645b052ecd1147" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40825 } }
I20260928 06:28:30.463276 25763 ts_tablet_manager.cc:1428] T c7b68fa203b14738a791526c243eb782 P 1f5037e17b564c78be31bd2014807c9e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 06:28:30.463300 25762 tablet_bootstrap.cc:654] T c7b68fa203b14738a791526c243eb782 P 533e2ce30a5e48c6be272da7a20dfd63: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:30.463928 25762 tablet_bootstrap.cc:492] T c7b68fa203b14738a791526c243eb782 P 533e2ce30a5e48c6be272da7a20dfd63: No bootstrap required, opened a new log
I20260928 06:28:30.464025 25762 ts_tablet_manager.cc:1397] T c7b68fa203b14738a791526c243eb782 P 533e2ce30a5e48c6be272da7a20dfd63: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:28:30.464151 25762 raft_consensus.cc:348] T c7b68fa203b14738a791526c243eb782 P 533e2ce30a5e48c6be272da7a20dfd63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5037e17b564c78be31bd2014807c9e" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 38889 } } peers { permanent_uuid: "533e2ce30a5e48c6be272da7a20dfd63" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46149 } } peers { permanent_uuid: "6f91fe4bd8464f3bad645b052ecd1147" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40825 } }
I20260928 06:28:30.464255 25762 raft_consensus.cc:374] T c7b68fa203b14738a791526c243eb782 P 533e2ce30a5e48c6be272da7a20dfd63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:30.464278 25762 raft_consensus.cc:729] T c7b68fa203b14738a791526c243eb782 P 533e2ce30a5e48c6be272da7a20dfd63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 533e2ce30a5e48c6be272da7a20dfd63, State: Initialized, Role: FOLLOWER
I20260928 06:28:30.464339 25762 consensus_queue.cc:260] T c7b68fa203b14738a791526c243eb782 P 533e2ce30a5e48c6be272da7a20dfd63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5037e17b564c78be31bd2014807c9e" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 38889 } } peers { permanent_uuid: "533e2ce30a5e48c6be272da7a20dfd63" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46149 } } peers { permanent_uuid: "6f91fe4bd8464f3bad645b052ecd1147" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40825 } }
I20260928 06:28:30.464427 25762 ts_tablet_manager.cc:1428] T c7b68fa203b14738a791526c243eb782 P 533e2ce30a5e48c6be272da7a20dfd63: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 06:28:30.790684 25728 raft_consensus.cc:482] T c7b68fa203b14738a791526c243eb782 P 533e2ce30a5e48c6be272da7a20dfd63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:28:30.790808 25728 raft_consensus.cc:504] T c7b68fa203b14738a791526c243eb782 P 533e2ce30a5e48c6be272da7a20dfd63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5037e17b564c78be31bd2014807c9e" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 38889 } } peers { permanent_uuid: "533e2ce30a5e48c6be272da7a20dfd63" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46149 } } peers { permanent_uuid: "6f91fe4bd8464f3bad645b052ecd1147" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40825 } }
I20260928 06:28:30.790988 25728 leader_election.cc:288] T c7b68fa203b14738a791526c243eb782 P 533e2ce30a5e48c6be272da7a20dfd63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f5037e17b564c78be31bd2014807c9e (127.22.225.129:38889), 6f91fe4bd8464f3bad645b052ecd1147 (127.22.225.131:40825)
I20260928 06:28:30.791250 25319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7b68fa203b14738a791526c243eb782" candidate_uuid: "533e2ce30a5e48c6be272da7a20dfd63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f5037e17b564c78be31bd2014807c9e" is_pre_election: true
I20260928 06:28:30.791283 25598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7b68fa203b14738a791526c243eb782" candidate_uuid: "533e2ce30a5e48c6be272da7a20dfd63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f91fe4bd8464f3bad645b052ecd1147" is_pre_election: true
I20260928 06:28:30.791378 25319 raft_consensus.cc:2454] T c7b68fa203b14738a791526c243eb782 P 1f5037e17b564c78be31bd2014807c9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 533e2ce30a5e48c6be272da7a20dfd63 in term 0.
I20260928 06:28:30.791390 25598 raft_consensus.cc:2454] T c7b68fa203b14738a791526c243eb782 P 6f91fe4bd8464f3bad645b052ecd1147 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 533e2ce30a5e48c6be272da7a20dfd63 in term 0.
I20260928 06:28:30.791579 25390 leader_election.cc:302] T c7b68fa203b14738a791526c243eb782 P 533e2ce30a5e48c6be272da7a20dfd63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f5037e17b564c78be31bd2014807c9e, 533e2ce30a5e48c6be272da7a20dfd63; no voters:
I20260928 06:28:30.791702 25728 raft_consensus.cc:2781] T c7b68fa203b14738a791526c243eb782 P 533e2ce30a5e48c6be272da7a20dfd63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:28:30.791749 25728 raft_consensus.cc:482] T c7b68fa203b14738a791526c243eb782 P 533e2ce30a5e48c6be272da7a20dfd63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:28:30.791778 25728 raft_consensus.cc:3037] T c7b68fa203b14738a791526c243eb782 P 533e2ce30a5e48c6be272da7a20dfd63 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:30.792501 25728 raft_consensus.cc:504] T c7b68fa203b14738a791526c243eb782 P 533e2ce30a5e48c6be272da7a20dfd63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5037e17b564c78be31bd2014807c9e" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 38889 } } peers { permanent_uuid: "533e2ce30a5e48c6be272da7a20dfd63" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46149 } } peers { permanent_uuid: "6f91fe4bd8464f3bad645b052ecd1147" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40825 } }
I20260928 06:28:30.792654 25728 leader_election.cc:288] T c7b68fa203b14738a791526c243eb782 P 533e2ce30a5e48c6be272da7a20dfd63 [CANDIDATE]: Term 1 election: Requested vote from peers 1f5037e17b564c78be31bd2014807c9e (127.22.225.129:38889), 6f91fe4bd8464f3bad645b052ecd1147 (127.22.225.131:40825)
I20260928 06:28:30.792820 25598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7b68fa203b14738a791526c243eb782" candidate_uuid: "533e2ce30a5e48c6be272da7a20dfd63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f91fe4bd8464f3bad645b052ecd1147"
I20260928 06:28:30.792901 25598 raft_consensus.cc:3037] T c7b68fa203b14738a791526c243eb782 P 6f91fe4bd8464f3bad645b052ecd1147 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:30.792893 25319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7b68fa203b14738a791526c243eb782" candidate_uuid: "533e2ce30a5e48c6be272da7a20dfd63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f5037e17b564c78be31bd2014807c9e"
I20260928 06:28:30.792979 25319 raft_consensus.cc:3037] T c7b68fa203b14738a791526c243eb782 P 1f5037e17b564c78be31bd2014807c9e [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:30.793587 25319 raft_consensus.cc:2454] T c7b68fa203b14738a791526c243eb782 P 1f5037e17b564c78be31bd2014807c9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 533e2ce30a5e48c6be272da7a20dfd63 in term 1.
I20260928 06:28:30.793691 25598 raft_consensus.cc:2454] T c7b68fa203b14738a791526c243eb782 P 6f91fe4bd8464f3bad645b052ecd1147 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 533e2ce30a5e48c6be272da7a20dfd63 in term 1.
I20260928 06:28:30.793773 25390 leader_election.cc:302] T c7b68fa203b14738a791526c243eb782 P 533e2ce30a5e48c6be272da7a20dfd63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f5037e17b564c78be31bd2014807c9e, 533e2ce30a5e48c6be272da7a20dfd63; no voters:
I20260928 06:28:30.793918 25728 raft_consensus.cc:2781] T c7b68fa203b14738a791526c243eb782 P 533e2ce30a5e48c6be272da7a20dfd63 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:28:30.793974 25728 raft_consensus.cc:686] T c7b68fa203b14738a791526c243eb782 P 533e2ce30a5e48c6be272da7a20dfd63 [term 1 LEADER]: Becoming Leader. State: Replica: 533e2ce30a5e48c6be272da7a20dfd63, State: Running, Role: LEADER
I20260928 06:28:30.794044 25728 consensus_queue.cc:237] T c7b68fa203b14738a791526c243eb782 P 533e2ce30a5e48c6be272da7a20dfd63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5037e17b564c78be31bd2014807c9e" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 38889 } } peers { permanent_uuid: "533e2ce30a5e48c6be272da7a20dfd63" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46149 } } peers { permanent_uuid: "6f91fe4bd8464f3bad645b052ecd1147" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40825 } }
I20260928 06:28:30.794828 25160 catalog_manager.cc:5495] T c7b68fa203b14738a791526c243eb782 P 533e2ce30a5e48c6be272da7a20dfd63 reported cstate change: term changed from 0 to 1, leader changed from <none> to 533e2ce30a5e48c6be272da7a20dfd63 (127.22.225.130). New cstate: current_term: 1 leader_uuid: "533e2ce30a5e48c6be272da7a20dfd63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5037e17b564c78be31bd2014807c9e" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 38889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "533e2ce30a5e48c6be272da7a20dfd63" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f91fe4bd8464f3bad645b052ecd1147" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40825 } health_report { overall_health: UNKNOWN } } }
I20260928 06:28:30.819433 25770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:30.816880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37536 (local address 127.22.225.130:46149)
0928 06:28:30.817073 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:30.817075 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:30.817081 (+ 6us) server_negotiation.cc:407] Connection header received
0928 06:28:30.817099 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:30.817100 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:30.817109 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:30.817178 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:28:30.817342 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:30.817836 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:30.818751 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:30.818960 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:30.819032 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:30.819182 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:28:30.819217 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:28:30.819260 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:30.819319 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 06:28:30.819342 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":103,"threads_started":1}
I20260928 06:28:30.821601 25598 raft_consensus.cc:1261] T c7b68fa203b14738a791526c243eb782 P 6f91fe4bd8464f3bad645b052ecd1147 [term 1 FOLLOWER]: Refusing update from remote peer 533e2ce30a5e48c6be272da7a20dfd63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:28:30.821640 25319 raft_consensus.cc:1261] T c7b68fa203b14738a791526c243eb782 P 1f5037e17b564c78be31bd2014807c9e [term 1 FOLLOWER]: Refusing update from remote peer 533e2ce30a5e48c6be272da7a20dfd63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:28:30.821841 25728 consensus_queue.cc:1035] T c7b68fa203b14738a791526c243eb782 P 533e2ce30a5e48c6be272da7a20dfd63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f91fe4bd8464f3bad645b052ecd1147" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:28:30.821961 25736 consensus_queue.cc:1035] T c7b68fa203b14738a791526c243eb782 P 533e2ce30a5e48c6be272da7a20dfd63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f5037e17b564c78be31bd2014807c9e" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:28:30.831805 25161 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:28:30.831924 25161 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:28:30.832516 25161 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9128f0ee0f504bed81b2d03646b98fd4: Sending DeleteTablet for 3 replicas of tablet c7b68fa203b14738a791526c243eb782
I20260928 06:28:30.832773 25432 tablet_service.cc:1514] Processing DeleteTablet for tablet c7b68fa203b14738a791526c243eb782 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:28:30 UTC) from {username='kudu', principal='kudu/127.22.225.190@KRBTEST.COM'} at 127.0.0.1:37518
I20260928 06:28:30.832856 25567 tablet_service.cc:1514] Processing DeleteTablet for tablet c7b68fa203b14738a791526c243eb782 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:28:30 UTC) from {username='kudu', principal='kudu/127.22.225.190@KRBTEST.COM'} at 127.0.0.1:43402
I20260928 06:28:30.832880 25750 tablet_replica.cc:330] T c7b68fa203b14738a791526c243eb782 P 533e2ce30a5e48c6be272da7a20dfd63: stopping tablet replica
I20260928 06:28:30.832912 25299 tablet_service.cc:1514] Processing DeleteTablet for tablet c7b68fa203b14738a791526c243eb782 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:28:30 UTC) from {username='kudu', principal='kudu/127.22.225.190@KRBTEST.COM'} at 127.0.0.1:56874
I20260928 06:28:30.832944 25750 raft_consensus.cc:2229] T c7b68fa203b14738a791526c243eb782 P 533e2ce30a5e48c6be272da7a20dfd63 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:28:30.832994 25750 raft_consensus.cc:2258] T c7b68fa203b14738a791526c243eb782 P 533e2ce30a5e48c6be272da7a20dfd63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:28:30.832998 25752 tablet_replica.cc:330] T c7b68fa203b14738a791526c243eb782 P 6f91fe4bd8464f3bad645b052ecd1147: stopping tablet replica
I20260928 06:28:30.833043 25752 raft_consensus.cc:2229] T c7b68fa203b14738a791526c243eb782 P 6f91fe4bd8464f3bad645b052ecd1147 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:28:30.833087 25752 raft_consensus.cc:2258] T c7b68fa203b14738a791526c243eb782 P 6f91fe4bd8464f3bad645b052ecd1147 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:28:30.833155 25751 tablet_replica.cc:330] T c7b68fa203b14738a791526c243eb782 P 1f5037e17b564c78be31bd2014807c9e: stopping tablet replica
I20260928 06:28:30.833206 25751 raft_consensus.cc:2229] T c7b68fa203b14738a791526c243eb782 P 1f5037e17b564c78be31bd2014807c9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:28:30.833242 25751 raft_consensus.cc:2258] T c7b68fa203b14738a791526c243eb782 P 1f5037e17b564c78be31bd2014807c9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:28:30.833263 25750 ts_tablet_manager.cc:1905] T c7b68fa203b14738a791526c243eb782 P 533e2ce30a5e48c6be272da7a20dfd63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:28:30.833562 25751 ts_tablet_manager.cc:1905] T c7b68fa203b14738a791526c243eb782 P 1f5037e17b564c78be31bd2014807c9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:28:30.833786 25752 ts_tablet_manager.cc:1905] T c7b68fa203b14738a791526c243eb782 P 6f91fe4bd8464f3bad645b052ecd1147: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:28:30.834913 25750 ts_tablet_manager.cc:1918] T c7b68fa203b14738a791526c243eb782 P 533e2ce30a5e48c6be272da7a20dfd63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:28:30.834960 25750 log.cc:1196] T c7b68fa203b14738a791526c243eb782 P 533e2ce30a5e48c6be272da7a20dfd63: Deleting WAL directory at /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/ts-1/wal/wals/c7b68fa203b14738a791526c243eb782
I20260928 06:28:30.835064 25751 ts_tablet_manager.cc:1918] T c7b68fa203b14738a791526c243eb782 P 1f5037e17b564c78be31bd2014807c9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:28:30.835115 25751 log.cc:1196] T c7b68fa203b14738a791526c243eb782 P 1f5037e17b564c78be31bd2014807c9e: Deleting WAL directory at /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/ts-0/wal/wals/c7b68fa203b14738a791526c243eb782
I20260928 06:28:30.835167 25750 ts_tablet_manager.cc:1939] T c7b68fa203b14738a791526c243eb782 P 533e2ce30a5e48c6be272da7a20dfd63: Deleting consensus metadata
I20260928 06:28:30.835330 25751 ts_tablet_manager.cc:1939] T c7b68fa203b14738a791526c243eb782 P 1f5037e17b564c78be31bd2014807c9e: Deleting consensus metadata
I20260928 06:28:30.835325 25752 ts_tablet_manager.cc:1918] T c7b68fa203b14738a791526c243eb782 P 6f91fe4bd8464f3bad645b052ecd1147: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:28:30.835371 25752 log.cc:1196] T c7b68fa203b14738a791526c243eb782 P 6f91fe4bd8464f3bad645b052ecd1147: Deleting WAL directory at /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/ts-2/wal/wals/c7b68fa203b14738a791526c243eb782
I20260928 06:28:30.835484 25147 catalog_manager.cc:4841] TS 533e2ce30a5e48c6be272da7a20dfd63 (127.22.225.130:46149): tablet c7b68fa203b14738a791526c243eb782 (table test-table [id=37da873defb541b88650bfcb4ce5804c]) successfully deleted
I20260928 06:28:30.835582 25752 ts_tablet_manager.cc:1939] T c7b68fa203b14738a791526c243eb782 P 6f91fe4bd8464f3bad645b052ecd1147: Deleting consensus metadata
I20260928 06:28:30.835685 25161 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43816:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 06:28:30.835817 25146 catalog_manager.cc:4841] TS 1f5037e17b564c78be31bd2014807c9e (127.22.225.129:38889): tablet c7b68fa203b14738a791526c243eb782 (table test-table [id=37da873defb541b88650bfcb4ce5804c]) successfully deleted
W20260928 06:28:30.835832 25161 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:28:30.836195 25146 catalog_manager.cc:4841] TS 6f91fe4bd8464f3bad645b052ecd1147 (127.22.225.131:40825): tablet c7b68fa203b14738a791526c243eb782 (table test-table [id=37da873defb541b88650bfcb4ce5804c]) successfully deleted
I20260928 06:28:30.837189 25299 tablet_service.cc:1467] Processing CreateTablet for tablet 2ed022cbb9614bfc87291afe701c8c10 (DEFAULT_TABLE table=test-table [id=d33045a5d2c94d20a37a7830e8463467]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:28:30.837189 25432 tablet_service.cc:1467] Processing CreateTablet for tablet 2ed022cbb9614bfc87291afe701c8c10 (DEFAULT_TABLE table=test-table [id=d33045a5d2c94d20a37a7830e8463467]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:28:30.837299 25432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ed022cbb9614bfc87291afe701c8c10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:30.837299 25299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ed022cbb9614bfc87291afe701c8c10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:30.838502 25763 tablet_bootstrap.cc:492] T 2ed022cbb9614bfc87291afe701c8c10 P 1f5037e17b564c78be31bd2014807c9e: Bootstrap starting.
I20260928 06:28:30.838837 25763 tablet_bootstrap.cc:654] T 2ed022cbb9614bfc87291afe701c8c10 P 1f5037e17b564c78be31bd2014807c9e: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:30.839140 25762 tablet_bootstrap.cc:492] T 2ed022cbb9614bfc87291afe701c8c10 P 533e2ce30a5e48c6be272da7a20dfd63: Bootstrap starting.
I20260928 06:28:30.839437 25762 tablet_bootstrap.cc:654] T 2ed022cbb9614bfc87291afe701c8c10 P 533e2ce30a5e48c6be272da7a20dfd63: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:30.839720 25567 tablet_service.cc:1467] Processing CreateTablet for tablet 2ed022cbb9614bfc87291afe701c8c10 (DEFAULT_TABLE table=test-table [id=d33045a5d2c94d20a37a7830e8463467]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:28:30.839841 25567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ed022cbb9614bfc87291afe701c8c10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:30.839967 25763 tablet_bootstrap.cc:492] T 2ed022cbb9614bfc87291afe701c8c10 P 1f5037e17b564c78be31bd2014807c9e: No bootstrap required, opened a new log
I20260928 06:28:30.840035 25763 ts_tablet_manager.cc:1397] T 2ed022cbb9614bfc87291afe701c8c10 P 1f5037e17b564c78be31bd2014807c9e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:28:30.840163 25763 raft_consensus.cc:348] T 2ed022cbb9614bfc87291afe701c8c10 P 1f5037e17b564c78be31bd2014807c9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5037e17b564c78be31bd2014807c9e" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 38889 } } peers { permanent_uuid: "533e2ce30a5e48c6be272da7a20dfd63" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46149 } } peers { permanent_uuid: "6f91fe4bd8464f3bad645b052ecd1147" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40825 } }
I20260928 06:28:30.840230 25763 raft_consensus.cc:374] T 2ed022cbb9614bfc87291afe701c8c10 P 1f5037e17b564c78be31bd2014807c9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:30.840255 25763 raft_consensus.cc:729] T 2ed022cbb9614bfc87291afe701c8c10 P 1f5037e17b564c78be31bd2014807c9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f5037e17b564c78be31bd2014807c9e, State: Initialized, Role: FOLLOWER
I20260928 06:28:30.840291 25763 consensus_queue.cc:260] T 2ed022cbb9614bfc87291afe701c8c10 P 1f5037e17b564c78be31bd2014807c9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5037e17b564c78be31bd2014807c9e" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 38889 } } peers { permanent_uuid: "533e2ce30a5e48c6be272da7a20dfd63" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46149 } } peers { permanent_uuid: "6f91fe4bd8464f3bad645b052ecd1147" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40825 } }
I20260928 06:28:30.840381 25763 ts_tablet_manager.cc:1428] T 2ed022cbb9614bfc87291afe701c8c10 P 1f5037e17b564c78be31bd2014807c9e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:30.840704 25762 tablet_bootstrap.cc:492] T 2ed022cbb9614bfc87291afe701c8c10 P 533e2ce30a5e48c6be272da7a20dfd63: No bootstrap required, opened a new log
I20260928 06:28:30.840757 25762 ts_tablet_manager.cc:1397] T 2ed022cbb9614bfc87291afe701c8c10 P 533e2ce30a5e48c6be272da7a20dfd63: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20260928 06:28:30.840875 25762 raft_consensus.cc:348] T 2ed022cbb9614bfc87291afe701c8c10 P 533e2ce30a5e48c6be272da7a20dfd63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5037e17b564c78be31bd2014807c9e" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 38889 } } peers { permanent_uuid: "533e2ce30a5e48c6be272da7a20dfd63" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46149 } } peers { permanent_uuid: "6f91fe4bd8464f3bad645b052ecd1147" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40825 } }
I20260928 06:28:30.840941 25762 raft_consensus.cc:374] T 2ed022cbb9614bfc87291afe701c8c10 P 533e2ce30a5e48c6be272da7a20dfd63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:30.840960 25762 raft_consensus.cc:729] T 2ed022cbb9614bfc87291afe701c8c10 P 533e2ce30a5e48c6be272da7a20dfd63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 533e2ce30a5e48c6be272da7a20dfd63, State: Initialized, Role: FOLLOWER
I20260928 06:28:30.841008 25762 consensus_queue.cc:260] T 2ed022cbb9614bfc87291afe701c8c10 P 533e2ce30a5e48c6be272da7a20dfd63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5037e17b564c78be31bd2014807c9e" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 38889 } } peers { permanent_uuid: "533e2ce30a5e48c6be272da7a20dfd63" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46149 } } peers { permanent_uuid: "6f91fe4bd8464f3bad645b052ecd1147" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40825 } }
I20260928 06:28:30.841104 25762 ts_tablet_manager.cc:1428] T 2ed022cbb9614bfc87291afe701c8c10 P 533e2ce30a5e48c6be272da7a20dfd63: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:30.841226 25761 tablet_bootstrap.cc:492] T 2ed022cbb9614bfc87291afe701c8c10 P 6f91fe4bd8464f3bad645b052ecd1147: Bootstrap starting.
I20260928 06:28:30.841663 25761 tablet_bootstrap.cc:654] T 2ed022cbb9614bfc87291afe701c8c10 P 6f91fe4bd8464f3bad645b052ecd1147: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:30.842279 25761 tablet_bootstrap.cc:492] T 2ed022cbb9614bfc87291afe701c8c10 P 6f91fe4bd8464f3bad645b052ecd1147: No bootstrap required, opened a new log
I20260928 06:28:30.842319 25761 ts_tablet_manager.cc:1397] T 2ed022cbb9614bfc87291afe701c8c10 P 6f91fe4bd8464f3bad645b052ecd1147: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:28:30.842460 25761 raft_consensus.cc:348] T 2ed022cbb9614bfc87291afe701c8c10 P 6f91fe4bd8464f3bad645b052ecd1147 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5037e17b564c78be31bd2014807c9e" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 38889 } } peers { permanent_uuid: "533e2ce30a5e48c6be272da7a20dfd63" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46149 } } peers { permanent_uuid: "6f91fe4bd8464f3bad645b052ecd1147" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40825 } }
I20260928 06:28:30.842519 25761 raft_consensus.cc:374] T 2ed022cbb9614bfc87291afe701c8c10 P 6f91fe4bd8464f3bad645b052ecd1147 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:30.842532 25761 raft_consensus.cc:729] T 2ed022cbb9614bfc87291afe701c8c10 P 6f91fe4bd8464f3bad645b052ecd1147 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f91fe4bd8464f3bad645b052ecd1147, State: Initialized, Role: FOLLOWER
I20260928 06:28:30.842579 25761 consensus_queue.cc:260] T 2ed022cbb9614bfc87291afe701c8c10 P 6f91fe4bd8464f3bad645b052ecd1147 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5037e17b564c78be31bd2014807c9e" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 38889 } } peers { permanent_uuid: "533e2ce30a5e48c6be272da7a20dfd63" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46149 } } peers { permanent_uuid: "6f91fe4bd8464f3bad645b052ecd1147" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40825 } }
I20260928 06:28:30.842674 25761 ts_tablet_manager.cc:1428] T 2ed022cbb9614bfc87291afe701c8c10 P 6f91fe4bd8464f3bad645b052ecd1147: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 06:28:31.064299 25766 raft_consensus.cc:482] T 2ed022cbb9614bfc87291afe701c8c10 P 6f91fe4bd8464f3bad645b052ecd1147 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:28:31.064415 25766 raft_consensus.cc:504] T 2ed022cbb9614bfc87291afe701c8c10 P 6f91fe4bd8464f3bad645b052ecd1147 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5037e17b564c78be31bd2014807c9e" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 38889 } } peers { permanent_uuid: "533e2ce30a5e48c6be272da7a20dfd63" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46149 } } peers { permanent_uuid: "6f91fe4bd8464f3bad645b052ecd1147" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40825 } }
I20260928 06:28:31.064848 25766 leader_election.cc:288] T 2ed022cbb9614bfc87291afe701c8c10 P 6f91fe4bd8464f3bad645b052ecd1147 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f5037e17b564c78be31bd2014807c9e (127.22.225.129:38889), 533e2ce30a5e48c6be272da7a20dfd63 (127.22.225.130:46149)
I20260928 06:28:31.068802 25778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:31.065026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.129:38889 (local address 127.22.225.131:45853)
0928 06:28:31.065184 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:31.065196 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:31.065250 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:31.065684 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:31.065685 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:31.065688 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:28:31.065768 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:31.065770 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:31.066747 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:31.066748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:31.067637 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:31.067639 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:31.068439 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:31.068440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:31.068537 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:31.068544 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 06:28:31.068626 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 06:28:31.068675 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":26,"thread_start_us":87,"threads_started":1}
I20260928 06:28:31.069123 25779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:31.065361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:45853 (local address 127.22.225.129:38889)
0928 06:28:31.065549 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:31.065551 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:31.065560 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:28:31.065584 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:31.065585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:31.065596 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:31.065836 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:28:31.065892 (+ 56us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:31.066620 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:31.067733 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:31.068347 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:31.068424 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:31.068432 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:31.068926 (+ 494us) server_negotiation.cc:299] Negotiation successful
0928 06:28:31.068982 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":55,"threads_started":1}
I20260928 06:28:31.069458 25319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ed022cbb9614bfc87291afe701c8c10" candidate_uuid: "6f91fe4bd8464f3bad645b052ecd1147" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f5037e17b564c78be31bd2014807c9e" is_pre_election: true
I20260928 06:28:31.069581 25319 raft_consensus.cc:2454] T 2ed022cbb9614bfc87291afe701c8c10 P 1f5037e17b564c78be31bd2014807c9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f91fe4bd8464f3bad645b052ecd1147 in term 0.
I20260928 06:28:31.069757 25530 leader_election.cc:302] T 2ed022cbb9614bfc87291afe701c8c10 P 6f91fe4bd8464f3bad645b052ecd1147 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f5037e17b564c78be31bd2014807c9e, 6f91fe4bd8464f3bad645b052ecd1147; no voters:
I20260928 06:28:31.069866 25770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:31.065465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:39463 (local address 127.22.225.130:46149)
0928 06:28:31.065542 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:31.065544 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:31.065553 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:28:31.065585 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:31.065586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:31.065596 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:31.065651 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:28:31.065868 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:31.066626 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:31.068819 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:31.069440 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:31.069517 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:31.069528 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:31.069729 (+ 201us) server_negotiation.cc:299] Negotiation successful
0928 06:28:31.069782 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 06:28:31.069913 25766 raft_consensus.cc:2781] T 2ed022cbb9614bfc87291afe701c8c10 P 6f91fe4bd8464f3bad645b052ecd1147 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:28:31.069947 25766 raft_consensus.cc:482] T 2ed022cbb9614bfc87291afe701c8c10 P 6f91fe4bd8464f3bad645b052ecd1147 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:28:31.069972 25766 raft_consensus.cc:3037] T 2ed022cbb9614bfc87291afe701c8c10 P 6f91fe4bd8464f3bad645b052ecd1147 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:31.070139 25777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:31.065048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.130:46149 (local address 127.22.225.131:39463)
0928 06:28:31.065259 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:31.065305 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:31.065380 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:31.065702 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:31.065703 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:31.065704 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:28:31.065769 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:31.065770 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:31.066722 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:31.066723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:31.068731 (+ 2008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:31.068734 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:31.069523 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:31.069524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:31.069607 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:31.069612 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:28:31.069986 (+ 374us) client_negotiation.cc:240] Negotiation successful
0928 06:28:31.070040 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":43,"thread_start_us":68,"threads_started":1}
I20260928 06:28:31.070487 25459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ed022cbb9614bfc87291afe701c8c10" candidate_uuid: "6f91fe4bd8464f3bad645b052ecd1147" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "533e2ce30a5e48c6be272da7a20dfd63" is_pre_election: true
I20260928 06:28:31.070593 25459 raft_consensus.cc:2454] T 2ed022cbb9614bfc87291afe701c8c10 P 533e2ce30a5e48c6be272da7a20dfd63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f91fe4bd8464f3bad645b052ecd1147 in term 0.
I20260928 06:28:31.071170 25766 raft_consensus.cc:504] T 2ed022cbb9614bfc87291afe701c8c10 P 6f91fe4bd8464f3bad645b052ecd1147 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5037e17b564c78be31bd2014807c9e" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 38889 } } peers { permanent_uuid: "533e2ce30a5e48c6be272da7a20dfd63" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46149 } } peers { permanent_uuid: "6f91fe4bd8464f3bad645b052ecd1147" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40825 } }
I20260928 06:28:31.071314 25766 leader_election.cc:288] T 2ed022cbb9614bfc87291afe701c8c10 P 6f91fe4bd8464f3bad645b052ecd1147 [CANDIDATE]: Term 1 election: Requested vote from peers 1f5037e17b564c78be31bd2014807c9e (127.22.225.129:38889), 533e2ce30a5e48c6be272da7a20dfd63 (127.22.225.130:46149)
I20260928 06:28:31.071467 25319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ed022cbb9614bfc87291afe701c8c10" candidate_uuid: "6f91fe4bd8464f3bad645b052ecd1147" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f5037e17b564c78be31bd2014807c9e"
I20260928 06:28:31.071482 25459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ed022cbb9614bfc87291afe701c8c10" candidate_uuid: "6f91fe4bd8464f3bad645b052ecd1147" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "533e2ce30a5e48c6be272da7a20dfd63"
I20260928 06:28:31.071548 25459 raft_consensus.cc:3037] T 2ed022cbb9614bfc87291afe701c8c10 P 533e2ce30a5e48c6be272da7a20dfd63 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:31.071548 25319 raft_consensus.cc:3037] T 2ed022cbb9614bfc87291afe701c8c10 P 1f5037e17b564c78be31bd2014807c9e [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:31.072291 25319 raft_consensus.cc:2454] T 2ed022cbb9614bfc87291afe701c8c10 P 1f5037e17b564c78be31bd2014807c9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f91fe4bd8464f3bad645b052ecd1147 in term 1.
I20260928 06:28:31.072291 25459 raft_consensus.cc:2454] T 2ed022cbb9614bfc87291afe701c8c10 P 533e2ce30a5e48c6be272da7a20dfd63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f91fe4bd8464f3bad645b052ecd1147 in term 1.
I20260928 06:28:31.072463 25531 leader_election.cc:302] T 2ed022cbb9614bfc87291afe701c8c10 P 6f91fe4bd8464f3bad645b052ecd1147 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 533e2ce30a5e48c6be272da7a20dfd63, 6f91fe4bd8464f3bad645b052ecd1147; no voters:
I20260928 06:28:31.072602 25766 raft_consensus.cc:2781] T 2ed022cbb9614bfc87291afe701c8c10 P 6f91fe4bd8464f3bad645b052ecd1147 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:28:31.072759 25766 raft_consensus.cc:686] T 2ed022cbb9614bfc87291afe701c8c10 P 6f91fe4bd8464f3bad645b052ecd1147 [term 1 LEADER]: Becoming Leader. State: Replica: 6f91fe4bd8464f3bad645b052ecd1147, State: Running, Role: LEADER
I20260928 06:28:31.072834 25766 consensus_queue.cc:237] T 2ed022cbb9614bfc87291afe701c8c10 P 6f91fe4bd8464f3bad645b052ecd1147 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5037e17b564c78be31bd2014807c9e" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 38889 } } peers { permanent_uuid: "533e2ce30a5e48c6be272da7a20dfd63" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46149 } } peers { permanent_uuid: "6f91fe4bd8464f3bad645b052ecd1147" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40825 } }
I20260928 06:28:31.073674 25160 catalog_manager.cc:5495] T 2ed022cbb9614bfc87291afe701c8c10 P 6f91fe4bd8464f3bad645b052ecd1147 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f91fe4bd8464f3bad645b052ecd1147 (127.22.225.131). New cstate: current_term: 1 leader_uuid: "6f91fe4bd8464f3bad645b052ecd1147" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5037e17b564c78be31bd2014807c9e" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 38889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "533e2ce30a5e48c6be272da7a20dfd63" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f91fe4bd8464f3bad645b052ecd1147" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 40825 } health_report { overall_health: HEALTHY } } }
I20260928 06:28:31.125591 25782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:31.123018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43406 (local address 127.22.225.131:40825)
0928 06:28:31.123198 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:31.123199 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:31.123207 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:28:31.123252 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:31.123252 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:31.123272 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:31.123346 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:28:31.123479 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:31.123960 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:31.124844 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:31.125075 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:31.125130 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:31.125281 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:28:31.125387 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:28:31.125438 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:31.125501 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 06:28:31.125516 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":118,"threads_started":1}
I20260928 06:28:31.130491 25770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:31.128196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:55601 (local address 127.22.225.130:46149)
0928 06:28:31.128247 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:31.128250 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:31.128259 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:28:31.128315 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:31.128316 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:31.128328 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:31.128388 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:28:31.128517 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:31.129100 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:31.129813 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:31.130038 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:31.130102 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:31.130259 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:28:31.130289 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:28:31.130336 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:31.130402 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 06:28:31.130425 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 06:28:31.130489 25785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:31.128053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.130:46149 (local address 127.22.225.131:55601)
0928 06:28:31.128220 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:31.128229 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:31.128271 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:31.128388 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:31.128389 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:31.128390 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:28:31.128453 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:31.128454 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:31.129189 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:31.129189 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:31.129731 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:31.129732 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:31.130127 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:31.130127 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:31.130171 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:31.130183 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:28:31.130349 (+ 166us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:28:31.130350 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 06:28:31.130383 (+ 33us) client_negotiation.cc:240] Negotiation successful
0928 06:28:31.130414 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":83,"threads_started":1}
I20260928 06:28:31.132664 25319 raft_consensus.cc:1261] T 2ed022cbb9614bfc87291afe701c8c10 P 1f5037e17b564c78be31bd2014807c9e [term 1 FOLLOWER]: Refusing update from remote peer 6f91fe4bd8464f3bad645b052ecd1147: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:28:31.132664 25459 raft_consensus.cc:1261] T 2ed022cbb9614bfc87291afe701c8c10 P 533e2ce30a5e48c6be272da7a20dfd63 [term 1 FOLLOWER]: Refusing update from remote peer 6f91fe4bd8464f3bad645b052ecd1147: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:28:31.132942 25780 consensus_queue.cc:1035] T 2ed022cbb9614bfc87291afe701c8c10 P 6f91fe4bd8464f3bad645b052ecd1147 [LEADER]: Connected to new peer: Peer: permanent_uuid: "533e2ce30a5e48c6be272da7a20dfd63" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:28:31.133054 25766 consensus_queue.cc:1035] T 2ed022cbb9614bfc87291afe701c8c10 P 6f91fe4bd8464f3bad645b052ecd1147 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f5037e17b564c78be31bd2014807c9e" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:28:31.140653 25782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:31.137847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:52637 (local address 127.22.225.131:40825)
0928 06:28:31.137898 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:31.137900 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:31.137934 (+ 34us) server_negotiation.cc:407] Connection header received
0928 06:28:31.138070 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:31.138071 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:31.138082 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:31.138139 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:28:31.138323 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:31.138922 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:31.140003 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:31.140225 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:31.140271 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:31.140413 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:28:31.140446 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:28:31.140481 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:31.140526 (+ 45us) server_negotiation.cc:299] Negotiation successful
0928 06:28:31.140541 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 06:28:31.140671 25789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:31.137717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.131:40825 (local address 127.22.225.130:52637)
0928 06:28:31.137876 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:31.137888 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:31.137947 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:31.138152 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:31.138153 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:31.138154 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:28:31.138227 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:31.138229 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:31.139003 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:31.139004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:31.139885 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:31.139888 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:31.140284 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:31.140285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:31.140337 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:31.140341 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:28:31.140484 (+ 143us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:28:31.140485 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 06:28:31.140524 (+ 39us) client_negotiation.cc:240] Negotiation successful
0928 06:28:31.140561 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":88,"threads_started":1}
W20260928 06:28:31.141336 25732 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 06:28:32.660676 25160 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:28:32.660796 25160 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:28:32.661728 25160 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9128f0ee0f504bed81b2d03646b98fd4: Sending DeleteTablet for 3 replicas of tablet 2ed022cbb9614bfc87291afe701c8c10
I20260928 06:28:32.661979 25432 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ed022cbb9614bfc87291afe701c8c10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:28:32 UTC) from {username='kudu', principal='kudu/127.22.225.190@KRBTEST.COM'} at 127.0.0.1:37518
I20260928 06:28:32.662042 25567 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ed022cbb9614bfc87291afe701c8c10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:28:32 UTC) from {username='kudu', principal='kudu/127.22.225.190@KRBTEST.COM'} at 127.0.0.1:43402
I20260928 06:28:32.662134 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 25234
I20260928 06:28:32.662171 25812 tablet_replica.cc:330] T 2ed022cbb9614bfc87291afe701c8c10 P 533e2ce30a5e48c6be272da7a20dfd63: stopping tablet replica
I20260928 06:28:32.662168 25299 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ed022cbb9614bfc87291afe701c8c10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:28:32 UTC) from {username='kudu', principal='kudu/127.22.225.190@KRBTEST.COM'} at 127.0.0.1:56874
I20260928 06:28:32.662846 25813 tablet_replica.cc:330] T 2ed022cbb9614bfc87291afe701c8c10 P 6f91fe4bd8464f3bad645b052ecd1147: stopping tablet replica
I20260928 06:28:32.662859 25812 raft_consensus.cc:2229] T 2ed022cbb9614bfc87291afe701c8c10 P 533e2ce30a5e48c6be272da7a20dfd63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:28:32.662901 25813 raft_consensus.cc:2229] T 2ed022cbb9614bfc87291afe701c8c10 P 6f91fe4bd8464f3bad645b052ecd1147 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:28:32.662916 25812 raft_consensus.cc:2258] T 2ed022cbb9614bfc87291afe701c8c10 P 533e2ce30a5e48c6be272da7a20dfd63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:28:32.662984 25813 raft_consensus.cc:2258] T 2ed022cbb9614bfc87291afe701c8c10 P 6f91fe4bd8464f3bad645b052ecd1147 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:28:32.663179 25812 ts_tablet_manager.cc:1905] T 2ed022cbb9614bfc87291afe701c8c10 P 533e2ce30a5e48c6be272da7a20dfd63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:28:32.663187 25813 ts_tablet_manager.cc:1905] T 2ed022cbb9614bfc87291afe701c8c10 P 6f91fe4bd8464f3bad645b052ecd1147: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:28:32.664860 25813 ts_tablet_manager.cc:1918] T 2ed022cbb9614bfc87291afe701c8c10 P 6f91fe4bd8464f3bad645b052ecd1147: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 06:28:32.664861 25812 ts_tablet_manager.cc:1918] T 2ed022cbb9614bfc87291afe701c8c10 P 533e2ce30a5e48c6be272da7a20dfd63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 06:28:32.664917 25813 log.cc:1196] T 2ed022cbb9614bfc87291afe701c8c10 P 6f91fe4bd8464f3bad645b052ecd1147: Deleting WAL directory at /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/ts-2/wal/wals/2ed022cbb9614bfc87291afe701c8c10
I20260928 06:28:32.664942 25812 log.cc:1196] T 2ed022cbb9614bfc87291afe701c8c10 P 533e2ce30a5e48c6be272da7a20dfd63: Deleting WAL directory at /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576899143097-23430-0/minicluster-data/ts-1/wal/wals/2ed022cbb9614bfc87291afe701c8c10
I20260928 06:28:32.665160 25812 ts_tablet_manager.cc:1939] T 2ed022cbb9614bfc87291afe701c8c10 P 533e2ce30a5e48c6be272da7a20dfd63: Deleting consensus metadata
I20260928 06:28:32.665190 25813 ts_tablet_manager.cc:1939] T 2ed022cbb9614bfc87291afe701c8c10 P 6f91fe4bd8464f3bad645b052ecd1147: Deleting consensus metadata
I20260928 06:28:32.665493 25147 catalog_manager.cc:4841] TS 533e2ce30a5e48c6be272da7a20dfd63 (127.22.225.130:46149): tablet 2ed022cbb9614bfc87291afe701c8c10 (table test-table [id=d33045a5d2c94d20a37a7830e8463467]) successfully deleted
I20260928 06:28:32.665504 25146 catalog_manager.cc:4841] TS 6f91fe4bd8464f3bad645b052ecd1147 (127.22.225.131:40825): tablet 2ed022cbb9614bfc87291afe701c8c10 (table test-table [id=d33045a5d2c94d20a37a7830e8463467]) successfully deleted
W20260928 06:28:32.670271 25146 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.225.129:38889 (error 108)
I20260928 06:28:32.670537 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 25374
I20260928 06:28:32.671489 25815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:32.670665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.129:38889 (local address 127.0.0.1:56886)
0928 06:28:32.671413 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:32.671455 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.129:38889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":724,"thread_start_us":56,"threads_started":1}
W20260928 06:28:32.671610 25146 catalog_manager.cc:4584] TS 1f5037e17b564c78be31bd2014807c9e (127.22.225.129:38889): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2ed022cbb9614bfc87291afe701c8c10: Network error: Client connection negotiation failed: client connection to 127.22.225.129:38889: connect: Connection refused (error 111)
I20260928 06:28:32.676942 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 25514
I20260928 06:28:32.683770 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 25129
2026-09-28T06:28:32Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6453 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:28:32 dist-test-slave-d59m krb5kdc[25818](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:28:32 dist-test-slave-d59m krb5kdc[25818](info): set up 2 sockets
Sep 28 06:28:32 dist-test-slave-d59m krb5kdc[25818](info): commencing operation
krb5kdc: starting...
W20260928 06:28:33.760946 23430 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Sep 28 06:28:33 dist-test-slave-d59m krb5kdc[25818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576913, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:28:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:28:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab.
Entry for principal kudu/127.22.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab.
I20260928 06:28:33.883680 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:46185
--webserver_interface=127.22.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab
--principal=kudu/127.22.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:37957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.225.190:46185
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:28:33.963953 25831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:28:33.964152 25831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:28:33.964174 25831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:28:33.965396 25831 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:28:33.965438 25831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:28:33.965453 25831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:28:33.965466 25831 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:28:33.965478 25831 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:28:33.966938 25831 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:37957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.225.190:46185
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab
--principal=kudu/127.22.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:46185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:28:33.967209 25831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:28:33.967453 25831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:28:33.973270 25837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:33.973335 25836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:33.973259 25831 server_base.cc:1034] running on GCE node
W20260928 06:28:33.973517 25839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:33.973784 25831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:28:33.974124 25831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:28:33.975292 25831 hybrid_clock.cc:648] HybridClock initialized: now 1790576913975270 us; error 43 us; skew 500 ppm
Sep 28 06:28:33 dist-test-slave-d59m krb5kdc[25818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576913, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:28:33.978168 25831 init.cc:377] Logged in from keytab as kudu/127.22.225.190@KRBTEST.COM (short username kudu)
I20260928 06:28:33.978641 25831 webserver.cc:460] Webserver started at http://127.22.225.190:35785/ using document root <none> and password file <none>
I20260928 06:28:33.978868 25831 fs_manager.cc:359] Metadata directory not provided
I20260928 06:28:33.978935 25831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:28:33.979096 25831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:28:33.980146 25831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/minicluster-data/master-0/data/instance:
uuid: "fd66702bfde54727b8a1ec07316d902f"
format_stamp: "Formatted at 2026-09-28 06:28:33 on dist-test-slave-d59m"
server_key: "e08608e97bfe0c13693013f9efc72e6a"
server_key_iv: "aaf6310c3b6c3dd480d4bb0271771a88"
server_key_version: "encryptionkey@0"
I20260928 06:28:33.980507 25831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/minicluster-data/master-0/wal/instance:
uuid: "fd66702bfde54727b8a1ec07316d902f"
format_stamp: "Formatted at 2026-09-28 06:28:33 on dist-test-slave-d59m"
server_key: "e08608e97bfe0c13693013f9efc72e6a"
server_key_iv: "aaf6310c3b6c3dd480d4bb0271771a88"
server_key_version: "encryptionkey@0"
I20260928 06:28:33.981879 25831 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:28:33.982621 25846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:33.982788 25831 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:28:33.982882 25831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/minicluster-data/master-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/minicluster-data/master-0/wal
uuid: "fd66702bfde54727b8a1ec07316d902f"
format_stamp: "Formatted at 2026-09-28 06:28:33 on dist-test-slave-d59m"
server_key: "e08608e97bfe0c13693013f9efc72e6a"
server_key_iv: "aaf6310c3b6c3dd480d4bb0271771a88"
server_key_version: "encryptionkey@0"
I20260928 06:28:33.982964 25831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:28:33.994994 25831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:28:33.997356 25831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:28:33.997524 25831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:28:34.002015 25831 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.190:46185
I20260928 06:28:34.002036 25898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.190:46185 every 8 connection(s)
I20260928 06:28:34.002437 25831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
I20260928 06:28:34.003402 25899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:34.006345 25899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd66702bfde54727b8a1ec07316d902f: Bootstrap starting.
I20260928 06:28:34.007009 25899 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd66702bfde54727b8a1ec07316d902f: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:34.007340 25899 log.cc:824] T 00000000000000000000000000000000 P fd66702bfde54727b8a1ec07316d902f: Log is configured to *not* fsync() on all Append() calls
I20260928 06:28:34.008069 25899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd66702bfde54727b8a1ec07316d902f: No bootstrap required, opened a new log
I20260928 06:28:34.008610 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 25831
I20260928 06:28:34.008726 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/minicluster-data/master-0/wal/instance
I20260928 06:28:34.008862 23430 external_mini_cluster.cc:1428] Setting key caac22c351d42639431a39d3c5ed0440
I20260928 06:28:34.010433 25899 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd66702bfde54727b8a1ec07316d902f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd66702bfde54727b8a1ec07316d902f" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 46185 } }
I20260928 06:28:34.010658 25899 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd66702bfde54727b8a1ec07316d902f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:34.010708 25899 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd66702bfde54727b8a1ec07316d902f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd66702bfde54727b8a1ec07316d902f, State: Initialized, Role: FOLLOWER
I20260928 06:28:34.010833 25899 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd66702bfde54727b8a1ec07316d902f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd66702bfde54727b8a1ec07316d902f" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 46185 } }
I20260928 06:28:34.010903 25899 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd66702bfde54727b8a1ec07316d902f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:28:34.010948 25899 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd66702bfde54727b8a1ec07316d902f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:28:34.011011 25899 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd66702bfde54727b8a1ec07316d902f [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:34.012050 25899 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd66702bfde54727b8a1ec07316d902f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd66702bfde54727b8a1ec07316d902f" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 46185 } }
I20260928 06:28:34.012153 25899 leader_election.cc:302] T 00000000000000000000000000000000 P fd66702bfde54727b8a1ec07316d902f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd66702bfde54727b8a1ec07316d902f; no voters:
I20260928 06:28:34.012329 25899 leader_election.cc:288] T 00000000000000000000000000000000 P fd66702bfde54727b8a1ec07316d902f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:28:34.012382 25904 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd66702bfde54727b8a1ec07316d902f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:28:34.012635 25904 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd66702bfde54727b8a1ec07316d902f [term 1 LEADER]: Becoming Leader. State: Replica: fd66702bfde54727b8a1ec07316d902f, State: Running, Role: LEADER
I20260928 06:28:34.012693 25899 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd66702bfde54727b8a1ec07316d902f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:28:34.013327 25904 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd66702bfde54727b8a1ec07316d902f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd66702bfde54727b8a1ec07316d902f" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 46185 } }
I20260928 06:28:34.013739 25905 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd66702bfde54727b8a1ec07316d902f [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd66702bfde54727b8a1ec07316d902f. Latest consensus state: current_term: 1 leader_uuid: "fd66702bfde54727b8a1ec07316d902f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd66702bfde54727b8a1ec07316d902f" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 46185 } } }
I20260928 06:28:34.013805 25905 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd66702bfde54727b8a1ec07316d902f [sys.catalog]: This master's current role is: LEADER
I20260928 06:28:34.014081 25906 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd66702bfde54727b8a1ec07316d902f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd66702bfde54727b8a1ec07316d902f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd66702bfde54727b8a1ec07316d902f" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 46185 } } }
I20260928 06:28:34.014127 25906 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd66702bfde54727b8a1ec07316d902f [sys.catalog]: This master's current role is: LEADER
I20260928 06:28:34.014360 25918 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:28:34.014834 25918 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 06:28:34 dist-test-slave-d59m krb5kdc[25818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576913, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:28:34.020787 25903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:34.009337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60518 (local address 127.22.225.190:46185)
0928 06:28:34.009516 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:34.009526 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:34.009561 (+ 35us) server_negotiation.cc:407] Connection header received
0928 06:28:34.010398 (+ 837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:34.010404 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:34.010456 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:34.010591 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:34.011356 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:34.012372 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:34.013570 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:34.014339 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:34.015607 (+ 1268us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:34.017396 (+ 1789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:34.017400 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:34.017400 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:34.017412 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:34.019205 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:34.019572 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:34.019575 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:34.019575 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:34.019700 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:34.019920 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:34.019920 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:34.019921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:34.020191 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:34.020365 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:34.020429 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 06:28:34.020591 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":70,"threads_started":1}
I20260928 06:28:34.025038 25918 catalog_manager.cc:1347] Generated new cluster ID: a05c36ac69f6401ab0af6cf65debdd79
I20260928 06:28:34.025102 25918 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:28:34.056385 25918 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:28:34.057127 25918 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:28:34.064043 25918 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd66702bfde54727b8a1ec07316d902f: Generated new TSK 0
I20260928 06:28:34.064229 25918 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.225.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab.
Entry for principal kudu/127.22.225.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab.
I20260928 06:28:34.096078 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.129:0
--local_ip_for_outbound_sockets=127.22.225.129
--webserver_interface=127.22.225.129
--webserver_port=0
--tserver_master_addrs=127.22.225.190:46185
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab
--principal=kudu/127.22.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:37957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:28:34.205168 25925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:28:34.205404 25925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:28:34.205453 25925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:28:34.206653 25925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:28:34.206720 25925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:28:34.206766 25925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.129
I20260928 06:28:34.208230 25925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:37957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab
--principal=kudu/127.22.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.225.129
--webserver_port=0
--tserver_master_addrs=127.22.225.190:46185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25925
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.129
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:28:34.208510 25925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:28:34.208765 25925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:28:34.213759 25933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:34.213789 25930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:34.213785 25931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:34.214170 25925 server_base.cc:1034] running on GCE node
I20260928 06:28:34.214382 25925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:28:34.214622 25925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:28:34.215791 25925 hybrid_clock.cc:648] HybridClock initialized: now 1790576914215778 us; error 29 us; skew 500 ppm
Sep 28 06:28:34 dist-test-slave-d59m krb5kdc[25818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576914, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:28:34.218541 25925 init.cc:377] Logged in from keytab as kudu/127.22.225.129@KRBTEST.COM (short username kudu)
I20260928 06:28:34.218982 25925 webserver.cc:460] Webserver started at http://127.22.225.129:37661/ using document root <none> and password file <none>
I20260928 06:28:34.219231 25925 fs_manager.cc:359] Metadata directory not provided
I20260928 06:28:34.219301 25925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:28:34.219442 25925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:28:34.220513 25925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-0/data/instance:
uuid: "f487587f107448aca23f765676824cad"
format_stamp: "Formatted at 2026-09-28 06:28:34 on dist-test-slave-d59m"
server_key: "c9dc71177195001e96d288bc02603aaa"
server_key_iv: "fb426a9797f864ed6b22c796d41073d7"
server_key_version: "encryptionkey@0"
I20260928 06:28:34.220958 25925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-0/wal/instance:
uuid: "f487587f107448aca23f765676824cad"
format_stamp: "Formatted at 2026-09-28 06:28:34 on dist-test-slave-d59m"
server_key: "c9dc71177195001e96d288bc02603aaa"
server_key_iv: "fb426a9797f864ed6b22c796d41073d7"
server_key_version: "encryptionkey@0"
I20260928 06:28:34.222321 25925 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:28:34.223138 25940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:34.223779 25925 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:28:34.223875 25925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-0/wal
uuid: "f487587f107448aca23f765676824cad"
format_stamp: "Formatted at 2026-09-28 06:28:34 on dist-test-slave-d59m"
server_key: "c9dc71177195001e96d288bc02603aaa"
server_key_iv: "fb426a9797f864ed6b22c796d41073d7"
server_key_version: "encryptionkey@0"
I20260928 06:28:34.223976 25925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:28:34.231469 25925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:28:34.233832 25925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:28:34.234000 25925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:28:34.234226 25925 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:28:34.234547 25925 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:28:34.234607 25925 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:34.234650 25925 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:28:34.234678 25925 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:34.240967 25925 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.129:40539
I20260928 06:28:34.241043 26053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.129:40539 every 8 connection(s)
I20260928 06:28:34.241370 25925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
I20260928 06:28:34.241923 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 25925
I20260928 06:28:34.242012 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-0/wal/instance
I20260928 06:28:34.242206 23430 external_mini_cluster.cc:1428] Setting key e3f65b3d5bbf2a34bcf8a296284a1080
Sep 28 06:28:34 dist-test-slave-d59m krb5kdc[25818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576914, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.129@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.225.130@KRBTEST.COM; defaulting to no policy
I20260928 06:28:34.252481 25903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:34.242254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:56379 (local address 127.22.225.190:46185)
0928 06:28:34.242302 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:34.242304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:34.242575 (+ 271us) server_negotiation.cc:407] Connection header received
0928 06:28:34.243197 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:34.243198 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:34.243210 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:34.243273 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:34.244571 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:34.245115 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:34.246750 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:34.247074 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:34.247153 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:34.249573 (+ 2420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:34.249575 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:34.249575 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:34.249583 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:34.251030 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:34.251437 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:34.251439 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:34.251440 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:34.251501 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:34.251793 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:34.251794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:34.251794 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:34.251925 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:34.251999 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:34.252300 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 06:28:34.252389 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:28:34.252790 26056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:34.242249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:46185 (local address 127.22.225.129:56379)
0928 06:28:34.242429 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:34.242477 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:34.243092 (+ 615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:34.243394 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:34.243399 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:34.244228 (+ 829us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:34.244470 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:34.244476 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:34.245225 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:34.245226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:34.246644 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:34.246647 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:34.247175 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:34.247177 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:34.247329 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:34.247918 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:34.247931 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:34.249469 (+ 1538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:34.251132 (+ 1663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:34.251139 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:34.251141 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:34.251359 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:34.251570 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:34.251571 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:34.251571 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:34.251701 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:34.252003 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:34.252009 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:34.252144 (+ 135us) client_negotiation.cc:769] Sending connection context
0928 06:28:34.252265 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 06:28:34.252423 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":91,"threads_started":1}
I20260928 06:28:34.253304 26054 heartbeater.cc:347] Connected to a master server at 127.22.225.190:46185
I20260928 06:28:34.253417 26054 heartbeater.cc:464] Registering TS with master...
I20260928 06:28:34.253618 26054 heartbeater.cc:511] Master 127.22.225.190:46185 requested a full tablet report, sending...
I20260928 06:28:34.254187 25863 ts_manager.cc:194] Registered new tserver with Master: f487587f107448aca23f765676824cad (127.22.225.129:40539)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.130@KRBTEST.COM" created.
I20260928 06:28:34.255000 25863 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.225.129@KRBTEST.COM'} at 127.22.225.129:56379
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab.
Entry for principal kudu/127.22.225.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab.
I20260928 06:28:34.270744 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.130:0
--local_ip_for_outbound_sockets=127.22.225.130
--webserver_interface=127.22.225.130
--webserver_port=0
--tserver_master_addrs=127.22.225.190:46185
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab
--principal=kudu/127.22.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:37957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:28:34.356870 26059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:28:34.357066 26059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:28:34.357089 26059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:28:34.358282 26059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:28:34.358328 26059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:28:34.358354 26059 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.130
I20260928 06:28:34.359779 26059 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:37957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab
--principal=kudu/127.22.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.225.130
--webserver_port=0
--tserver_master_addrs=127.22.225.190:46185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.130
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:28:34.360034 26059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:28:34.360271 26059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:28:34.365154 26067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:34.365176 26065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:34.365378 26059 server_base.cc:1034] running on GCE node
W20260928 06:28:34.365152 26064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:34.365691 26059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:28:34.366006 26059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:28:34.367192 26059 hybrid_clock.cc:648] HybridClock initialized: now 1790576914367175 us; error 36 us; skew 500 ppm
Sep 28 06:28:34 dist-test-slave-d59m krb5kdc[25818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576914, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:28:34.370312 26059 init.cc:377] Logged in from keytab as kudu/127.22.225.130@KRBTEST.COM (short username kudu)
I20260928 06:28:34.370868 26059 webserver.cc:460] Webserver started at http://127.22.225.130:41407/ using document root <none> and password file <none>
I20260928 06:28:34.371150 26059 fs_manager.cc:359] Metadata directory not provided
I20260928 06:28:34.371222 26059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:28:34.371361 26059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:28:34.372668 26059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-1/data/instance:
uuid: "f6148fa58b2a443fbbc0dfc91d65d234"
format_stamp: "Formatted at 2026-09-28 06:28:34 on dist-test-slave-d59m"
server_key: "0d8a115b116f5d3b06b3a63f539516f4"
server_key_iv: "8282fd5b2cd842ffde33025cbb64d9e8"
server_key_version: "encryptionkey@0"
I20260928 06:28:34.373050 26059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-1/wal/instance:
uuid: "f6148fa58b2a443fbbc0dfc91d65d234"
format_stamp: "Formatted at 2026-09-28 06:28:34 on dist-test-slave-d59m"
server_key: "0d8a115b116f5d3b06b3a63f539516f4"
server_key_iv: "8282fd5b2cd842ffde33025cbb64d9e8"
server_key_version: "encryptionkey@0"
I20260928 06:28:34.374567 26059 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:28:34.375366 26074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:34.375559 26059 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:28:34.375679 26059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-1/wal
uuid: "f6148fa58b2a443fbbc0dfc91d65d234"
format_stamp: "Formatted at 2026-09-28 06:28:34 on dist-test-slave-d59m"
server_key: "0d8a115b116f5d3b06b3a63f539516f4"
server_key_iv: "8282fd5b2cd842ffde33025cbb64d9e8"
server_key_version: "encryptionkey@0"
I20260928 06:28:34.375759 26059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:28:34.383594 26059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:28:34.386020 26059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:28:34.386179 26059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:28:34.386407 26059 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:28:34.386760 26059 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:28:34.386812 26059 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:34.386853 26059 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:28:34.386880 26059 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:34.392347 26059 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.130:36313
I20260928 06:28:34.392402 26187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.130:36313 every 8 connection(s)
I20260928 06:28:34.392799 26059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
I20260928 06:28:34.396075 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 26059
I20260928 06:28:34.396167 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-1/wal/instance
I20260928 06:28:34.396353 23430 external_mini_cluster.cc:1428] Setting key 27a03b713b4577112c998c1579bf3cde
Sep 28 06:28:34 dist-test-slave-d59m krb5kdc[25818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576914, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.130@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:28:34.403375 25903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:34.393564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:57669 (local address 127.22.225.190:46185)
0928 06:28:34.393632 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:34.393635 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:34.393878 (+ 243us) server_negotiation.cc:407] Connection header received
0928 06:28:34.394496 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:34.394497 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:34.394507 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:34.394552 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:34.395709 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:34.396279 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:34.397466 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:34.397673 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:34.397759 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:34.400214 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:34.400215 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:34.400216 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:34.400223 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:34.402056 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:34.402431 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:34.402433 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:34.402433 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:34.402502 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:34.402732 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:34.402733 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:34.402733 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:34.402873 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:34.402948 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:34.403180 (+ 232us) server_negotiation.cc:299] Negotiation successful
0928 06:28:34.403266 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 06:28:34.403406 26190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:34.393549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:46185 (local address 127.22.225.130:57669)
0928 06:28:34.393738 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:34.393780 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:34.394385 (+ 605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:34.394624 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:34.394628 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:34.395362 (+ 734us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:34.395624 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:34.395629 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:34.396370 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:34.396371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:34.397341 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:34.397344 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:34.397768 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:34.397769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:34.397888 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:34.398417 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:34.398425 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:34.400120 (+ 1695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:34.402136 (+ 2016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:34.402141 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:34.402143 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:34.402356 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:34.402574 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:34.402574 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:34.402574 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:34.402666 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:34.402940 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:34.402944 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:34.403054 (+ 110us) client_negotiation.cc:769] Sending connection context
0928 06:28:34.403133 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 06:28:34.403249 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":101,"threads_started":1}
WARNING: no policy specified for kudu/127.22.225.131@KRBTEST.COM; defaulting to no policy
I20260928 06:28:34.406451 26188 heartbeater.cc:347] Connected to a master server at 127.22.225.190:46185
I20260928 06:28:34.406572 26188 heartbeater.cc:464] Registering TS with master...
I20260928 06:28:34.406786 26188 heartbeater.cc:511] Master 127.22.225.190:46185 requested a full tablet report, sending...
I20260928 06:28:34.407281 25863 ts_manager.cc:194] Registered new tserver with Master: f6148fa58b2a443fbbc0dfc91d65d234 (127.22.225.130:36313)
I20260928 06:28:34.407990 25863 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.225.130@KRBTEST.COM'} at 127.22.225.130:57669
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab.
Entry for principal kudu/127.22.225.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab.
I20260928 06:28:34.423624 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.131:0
--local_ip_for_outbound_sockets=127.22.225.131
--webserver_interface=127.22.225.131
--webserver_port=0
--tserver_master_addrs=127.22.225.190:46185
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab
--principal=kudu/127.22.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:37957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:28:34.512251 26193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:28:34.512554 26193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:28:34.512607 26193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:28:34.514539 26193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:28:34.514603 26193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:28:34.514648 26193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.131
I20260928 06:28:34.517211 26193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:37957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab
--principal=kudu/127.22.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.225.131
--webserver_port=0
--tserver_master_addrs=127.22.225.190:46185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.131
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:28:34.517475 26193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:28:34.517731 26193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:28:34.523362 26201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:34.523416 26199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:34.523660 26198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:34.523727 26193 server_base.cc:1034] running on GCE node
I20260928 06:28:34.523977 26193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:28:34.524271 26193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:28:34.525456 26193 hybrid_clock.cc:648] HybridClock initialized: now 1790576914525440 us; error 27 us; skew 500 ppm
Sep 28 06:28:34 dist-test-slave-d59m krb5kdc[25818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576914, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:28:34.528780 26193 init.cc:377] Logged in from keytab as kudu/127.22.225.131@KRBTEST.COM (short username kudu)
I20260928 06:28:34.529325 26193 webserver.cc:460] Webserver started at http://127.22.225.131:41647/ using document root <none> and password file <none>
I20260928 06:28:34.529608 26193 fs_manager.cc:359] Metadata directory not provided
I20260928 06:28:34.529666 26193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:28:34.529807 26193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:28:34.531272 26193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-2/data/instance:
uuid: "7a87d3cb5f4c40b0963a411b89e7ee06"
format_stamp: "Formatted at 2026-09-28 06:28:34 on dist-test-slave-d59m"
server_key: "611649bb082bc262ab1f5523972d8f92"
server_key_iv: "482fc2daa0a6e916e6ef19b8d1020c56"
server_key_version: "encryptionkey@0"
I20260928 06:28:34.531724 26193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-2/wal/instance:
uuid: "7a87d3cb5f4c40b0963a411b89e7ee06"
format_stamp: "Formatted at 2026-09-28 06:28:34 on dist-test-slave-d59m"
server_key: "611649bb082bc262ab1f5523972d8f92"
server_key_iv: "482fc2daa0a6e916e6ef19b8d1020c56"
server_key_version: "encryptionkey@0"
I20260928 06:28:34.533514 26193 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260928 06:28:34.534436 26208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:34.534632 26193 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:28:34.534735 26193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-2/wal
uuid: "7a87d3cb5f4c40b0963a411b89e7ee06"
format_stamp: "Formatted at 2026-09-28 06:28:34 on dist-test-slave-d59m"
server_key: "611649bb082bc262ab1f5523972d8f92"
server_key_iv: "482fc2daa0a6e916e6ef19b8d1020c56"
server_key_version: "encryptionkey@0"
I20260928 06:28:34.534821 26193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:28:34.551977 26193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:28:34.554113 26193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:28:34.554287 26193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:28:34.554534 26193 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:28:34.554857 26193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:28:34.554912 26193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:34.554957 26193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:28:34.554981 26193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:34.560827 26193 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.131:44791
I20260928 06:28:34.560865 26321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.131:44791 every 8 connection(s)
I20260928 06:28:34.561228 26193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
Sep 28 06:28:34 dist-test-slave-d59m krb5kdc[25818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576914, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.131@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:28:34.568791 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 26193
I20260928 06:28:34.568895 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-2/wal/instance
I20260928 06:28:34.569101 23430 external_mini_cluster.cc:1428] Setting key 4b3c63912201e84881357f09bd07a5b8
I20260928 06:28:34.570699 25903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:34.562066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:53273 (local address 127.22.225.190:46185)
0928 06:28:34.562142 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:34.562144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:34.562305 (+ 161us) server_negotiation.cc:407] Connection header received
0928 06:28:34.562762 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:34.562763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:34.562773 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:34.562826 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:34.563697 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:34.564322 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:34.565598 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:34.565771 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:34.565826 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:34.567917 (+ 2091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:34.567919 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:34.567919 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:34.567926 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:34.569422 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:34.569909 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:34.569911 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:34.569911 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:34.569974 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:34.570235 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:34.570235 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:34.570236 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:34.570336 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:34.570386 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:34.570580 (+ 194us) server_negotiation.cc:299] Negotiation successful
0928 06:28:34.570638 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 06:28:34.570892 26324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:34.562020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:46185 (local address 127.22.225.131:53273)
0928 06:28:34.562190 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:34.562226 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:34.562685 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:34.562889 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:34.562892 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:34.563431 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:34.563625 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:34.563629 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:34.564422 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:34.564423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:34.565519 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:34.565522 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:34.565897 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:34.565899 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:34.566000 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:34.566449 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:34.566457 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:34.567832 (+ 1375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:34.569525 (+ 1693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:34.569531 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:34.569534 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:34.569833 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:34.570057 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:34.570058 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:34.570059 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:34.570177 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:34.570377 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:34.570380 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:34.570484 (+ 104us) client_negotiation.cc:769] Sending connection context
0928 06:28:34.570561 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 06:28:34.570686 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":98,"threads_started":1}
I20260928 06:28:34.571321 26322 heartbeater.cc:347] Connected to a master server at 127.22.225.190:46185
I20260928 06:28:34.571434 26322 heartbeater.cc:464] Registering TS with master...
I20260928 06:28:34.571609 26322 heartbeater.cc:511] Master 127.22.225.190:46185 requested a full tablet report, sending...
I20260928 06:28:34.571966 25863 ts_manager.cc:194] Registered new tserver with Master: 7a87d3cb5f4c40b0963a411b89e7ee06 (127.22.225.131:44791)
I20260928 06:28:34.572283 25863 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.225.131@KRBTEST.COM'} at 127.22.225.131:53273
I20260928 06:28:34.581005 23430 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 06:28:34.591706 25903 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:28:34.590867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60522 (local address 127.22.225.190:46185)
0928 06:28:34.590944 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:34.590946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:34.591022 (+ 76us) server_negotiation.cc:407] Connection header received
0928 06:28:34.591064 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:34.591065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:34.591074 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:34.591123 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:34.591639 (+ 516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60522: BlockingRecv error: recv got EOF from 127.0.0.1:60522 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 06:28:34.592561 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 25925
I20260928 06:28:34.597170 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 26059
I20260928 06:28:34.601704 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 26193
I20260928 06:28:34.606416 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 25831
2026-09-28T06:28:34Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1922 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790576899143097-23430-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:28:34 dist-test-slave-d59m krb5kdc[26334](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:28:34 dist-test-slave-d59m krb5kdc[26334](info): set up 2 sockets
Sep 28 06:28:34 dist-test-slave-d59m krb5kdc[26334](info): commencing operation
krb5kdc: starting...
W20260928 06:28:35.676904 23430 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Sep 28 06:28:35 dist-test-slave-d59m krb5kdc[26334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:28:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:28:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab.
Entry for principal kudu/127.22.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab.
I20260928 06:28:35.795234 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790576899143097-23430-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790576899143097-23430-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790576899143097-23430-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:46021
--webserver_interface=127.22.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab
--principal=kudu/127.22.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:39191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.225.190:46021
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:28:35.880432 26347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:28:35.880699 26347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:28:35.880741 26347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:28:35.881865 26347 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:28:35.881930 26347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:28:35.881953 26347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:28:35.881974 26347 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:28:35.882009 26347 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:28:35.883458 26347 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:39191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790576899143097-23430-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790576899143097-23430-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.225.190:46021
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab
--principal=kudu/127.22.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:46021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790576899143097-23430-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:28:35.883718 26347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:28:35.883934 26347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:28:35.888703 26353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:35.888813 26355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:35.888746 26347 server_base.cc:1034] running on GCE node
W20260928 06:28:35.888707 26352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:35.889186 26347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:28:35.889532 26347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:28:35.890705 26347 hybrid_clock.cc:648] HybridClock initialized: now 1790576915890679 us; error 40 us; skew 500 ppm
Sep 28 06:28:35 dist-test-slave-d59m krb5kdc[26334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576915, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:28:35.893618 26347 init.cc:377] Logged in from keytab as kudu/127.22.225.190@KRBTEST.COM (short username kudu)
I20260928 06:28:35.894099 26347 webserver.cc:460] Webserver started at http://127.22.225.190:41283/ using document root <none> and password file <none>
I20260928 06:28:35.894321 26347 fs_manager.cc:359] Metadata directory not provided
I20260928 06:28:35.894387 26347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:28:35.894539 26347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:28:35.895584 26347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790576899143097-23430-0/minicluster-data/master-0/data/instance:
uuid: "e80649fae617409685e11cc1042a36ab"
format_stamp: "Formatted at 2026-09-28 06:28:35 on dist-test-slave-d59m"
server_key: "741fa639f6ab011f9eddcaae865204f8"
server_key_iv: "1dd4b49faad852522660f59c26d99725"
server_key_version: "encryptionkey@0"
I20260928 06:28:35.895941 26347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790576899143097-23430-0/minicluster-data/master-0/wal/instance:
uuid: "e80649fae617409685e11cc1042a36ab"
format_stamp: "Formatted at 2026-09-28 06:28:35 on dist-test-slave-d59m"
server_key: "741fa639f6ab011f9eddcaae865204f8"
server_key_iv: "1dd4b49faad852522660f59c26d99725"
server_key_version: "encryptionkey@0"
I20260928 06:28:35.897365 26347 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:28:35.898172 26362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:35.898370 26347 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:28:35.898468 26347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790576899143097-23430-0/minicluster-data/master-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790576899143097-23430-0/minicluster-data/master-0/wal
uuid: "e80649fae617409685e11cc1042a36ab"
format_stamp: "Formatted at 2026-09-28 06:28:35 on dist-test-slave-d59m"
server_key: "741fa639f6ab011f9eddcaae865204f8"
server_key_iv: "1dd4b49faad852522660f59c26d99725"
server_key_version: "encryptionkey@0"
I20260928 06:28:35.898550 26347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790576899143097-23430-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790576899143097-23430-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790576899143097-23430-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:28:35.917136 26347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:28:35.919735 26347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:28:35.919904 26347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:28:35.925196 26347 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.190:46021
I20260928 06:28:35.925220 26414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.190:46021 every 8 connection(s)
I20260928 06:28:35.925868 26347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
I20260928 06:28:35.926550 26415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:35.928953 26415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e80649fae617409685e11cc1042a36ab: Bootstrap starting.
I20260928 06:28:35.929543 26415 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e80649fae617409685e11cc1042a36ab: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:35.929828 26415 log.cc:824] T 00000000000000000000000000000000 P e80649fae617409685e11cc1042a36ab: Log is configured to *not* fsync() on all Append() calls
I20260928 06:28:35.930702 26415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e80649fae617409685e11cc1042a36ab: No bootstrap required, opened a new log
I20260928 06:28:35.930835 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 26347
I20260928 06:28:35.930976 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790576899143097-23430-0/minicluster-data/master-0/wal/instance
I20260928 06:28:35.931154 23430 external_mini_cluster.cc:1428] Setting key 5e358c13dc812b35b4f7e084ac782ed2
I20260928 06:28:35.934037 26415 raft_consensus.cc:348] T 00000000000000000000000000000000 P e80649fae617409685e11cc1042a36ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e80649fae617409685e11cc1042a36ab" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 46021 } }
I20260928 06:28:35.934218 26415 raft_consensus.cc:374] T 00000000000000000000000000000000 P e80649fae617409685e11cc1042a36ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:35.934252 26415 raft_consensus.cc:729] T 00000000000000000000000000000000 P e80649fae617409685e11cc1042a36ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e80649fae617409685e11cc1042a36ab, State: Initialized, Role: FOLLOWER
I20260928 06:28:35.934407 26415 consensus_queue.cc:260] T 00000000000000000000000000000000 P e80649fae617409685e11cc1042a36ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e80649fae617409685e11cc1042a36ab" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 46021 } }
I20260928 06:28:35.934494 26415 raft_consensus.cc:388] T 00000000000000000000000000000000 P e80649fae617409685e11cc1042a36ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:28:35.934525 26415 raft_consensus.cc:482] T 00000000000000000000000000000000 P e80649fae617409685e11cc1042a36ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:28:35.934588 26415 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e80649fae617409685e11cc1042a36ab [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:35.935259 26415 raft_consensus.cc:504] T 00000000000000000000000000000000 P e80649fae617409685e11cc1042a36ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e80649fae617409685e11cc1042a36ab" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 46021 } }
I20260928 06:28:35.935366 26415 leader_election.cc:302] T 00000000000000000000000000000000 P e80649fae617409685e11cc1042a36ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e80649fae617409685e11cc1042a36ab; no voters:
I20260928 06:28:35.935580 26415 leader_election.cc:288] T 00000000000000000000000000000000 P e80649fae617409685e11cc1042a36ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:28:35.935627 26420 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e80649fae617409685e11cc1042a36ab [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:28:35.935845 26420 raft_consensus.cc:686] T 00000000000000000000000000000000 P e80649fae617409685e11cc1042a36ab [term 1 LEADER]: Becoming Leader. State: Replica: e80649fae617409685e11cc1042a36ab, State: Running, Role: LEADER
I20260928 06:28:35.935886 26415 sys_catalog.cc:564] T 00000000000000000000000000000000 P e80649fae617409685e11cc1042a36ab [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:28:35.935997 26420 consensus_queue.cc:237] T 00000000000000000000000000000000 P e80649fae617409685e11cc1042a36ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e80649fae617409685e11cc1042a36ab" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 46021 } }
I20260928 06:28:35.936421 26420 sys_catalog.cc:455] T 00000000000000000000000000000000 P e80649fae617409685e11cc1042a36ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader e80649fae617409685e11cc1042a36ab. Latest consensus state: current_term: 1 leader_uuid: "e80649fae617409685e11cc1042a36ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e80649fae617409685e11cc1042a36ab" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 46021 } } }
I20260928 06:28:35.936609 26420 sys_catalog.cc:458] T 00000000000000000000000000000000 P e80649fae617409685e11cc1042a36ab [sys.catalog]: This master's current role is: LEADER
I20260928 06:28:35.936431 26421 sys_catalog.cc:455] T 00000000000000000000000000000000 P e80649fae617409685e11cc1042a36ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e80649fae617409685e11cc1042a36ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e80649fae617409685e11cc1042a36ab" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 46021 } } }
I20260928 06:28:35.936996 26421 sys_catalog.cc:458] T 00000000000000000000000000000000 P e80649fae617409685e11cc1042a36ab [sys.catalog]: This master's current role is: LEADER
I20260928 06:28:35.937031 26429 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:28:35.937567 26429 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:28:35.939889 26429 catalog_manager.cc:1347] Generated new cluster ID: 0bbfc65705874beb871f676755fac297
I20260928 06:28:35.939935 26429 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:28:35 dist-test-slave-d59m krb5kdc[26334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:28:35.948522 26419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:35.931678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36994 (local address 127.22.225.190:46021)
0928 06:28:35.931888 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:35.931898 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:35.931931 (+ 33us) server_negotiation.cc:407] Connection header received
0928 06:28:35.932456 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:35.932463 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:35.932512 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:35.932683 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:35.933415 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:35.934313 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:35.935174 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:35.935765 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:35.936173 (+ 408us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:35.940754 (+ 4581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:35.940758 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:35.940759 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:35.940770 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:35.946524 (+ 5754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:35.946942 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:35.946945 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:35.946945 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:35.947061 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:35.947254 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:35.947255 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:35.947255 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:35.947561 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:35.947709 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:35.948154 (+ 445us) server_negotiation.cc:299] Negotiation successful
0928 06:28:35.948292 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":93,"threads_started":1}
I20260928 06:28:35.949503 23430 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:28:35.953361 26429 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:28:35.954103 26429 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:28:35.958832 26429 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e80649fae617409685e11cc1042a36ab: Generated new TSK 0
I20260928 06:28:35.959082 26429 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:28:35.968026 26419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:35.964663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37010 (local address 127.22.225.190:46021)
0928 06:28:35.964699 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:35.964701 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:35.964709 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:28:35.964743 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:35.964744 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:35.964752 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:35.964842 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:35.965218 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:35.965728 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:35.966502 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:35.966734 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:35.966807 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:35.967550 (+ 743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:35.967552 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:35.967552 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:28:35.967745 (+ 193us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:35.967761 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:28:35.967767 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:35.967832 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:35.967896 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 06:28:35.967922 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 06:28:35.970400 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 26347
2026-09-28T06:28:35Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1360 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:28:36 dist-test-slave-d59m krb5kdc[26448](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:28:36 dist-test-slave-d59m krb5kdc[26448](info): set up 2 sockets
Sep 28 06:28:36 dist-test-slave-d59m krb5kdc[26448](info): commencing operation
krb5kdc: starting...
W20260928 06:28:37.089129 23430 mini_kdc.cc:121] Time spent starting KDC: real 1.100s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Sep 28 06:28:37 dist-test-slave-d59m krb5kdc[26448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:28:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:28:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab.
Entry for principal kudu/127.22.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab.
I20260928 06:28:37.215591 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:44963
--webserver_interface=127.22.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab
--principal=kudu/127.22.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:38425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.225.190:44963
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:28:37.297341 26461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:28:37.297587 26461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:28:37.297626 26461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:28:37.298888 26461 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:28:37.298954 26461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:28:37.298977 26461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:28:37.299005 26461 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:28:37.299031 26461 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:28:37.300468 26461 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:38425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.225.190:44963
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab
--principal=kudu/127.22.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:44963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:28:37.300760 26461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:28:37.300987 26461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:28:37.305801 26467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:37.305769 26469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:37.305787 26461 server_base.cc:1034] running on GCE node
W20260928 06:28:37.305907 26466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:37.306324 26461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:28:37.306648 26461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:28:37.307801 26461 hybrid_clock.cc:648] HybridClock initialized: now 1790576917307779 us; error 37 us; skew 500 ppm
Sep 28 06:28:37 dist-test-slave-d59m krb5kdc[26448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576917, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:28:37.310433 26461 init.cc:377] Logged in from keytab as kudu/127.22.225.190@KRBTEST.COM (short username kudu)
I20260928 06:28:37.310935 26461 webserver.cc:460] Webserver started at http://127.22.225.190:40097/ using document root <none> and password file <none>
I20260928 06:28:37.311180 26461 fs_manager.cc:359] Metadata directory not provided
I20260928 06:28:37.311239 26461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:28:37.311362 26461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:28:37.312670 26461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/minicluster-data/master-0/data/instance:
uuid: "57c34d143d78467e8579f5efa006e79d"
format_stamp: "Formatted at 2026-09-28 06:28:37 on dist-test-slave-d59m"
server_key: "65d01bff4a6392e505ec2d251bd3ff6c"
server_key_iv: "944cc85ae1c2f1bcc1af59d6e7bea7a8"
server_key_version: "encryptionkey@0"
I20260928 06:28:37.313067 26461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/minicluster-data/master-0/wal/instance:
uuid: "57c34d143d78467e8579f5efa006e79d"
format_stamp: "Formatted at 2026-09-28 06:28:37 on dist-test-slave-d59m"
server_key: "65d01bff4a6392e505ec2d251bd3ff6c"
server_key_iv: "944cc85ae1c2f1bcc1af59d6e7bea7a8"
server_key_version: "encryptionkey@0"
I20260928 06:28:37.314625 26461 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:28:37.315438 26476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:37.315668 26461 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:28:37.315774 26461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/minicluster-data/master-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/minicluster-data/master-0/wal
uuid: "57c34d143d78467e8579f5efa006e79d"
format_stamp: "Formatted at 2026-09-28 06:28:37 on dist-test-slave-d59m"
server_key: "65d01bff4a6392e505ec2d251bd3ff6c"
server_key_iv: "944cc85ae1c2f1bcc1af59d6e7bea7a8"
server_key_version: "encryptionkey@0"
I20260928 06:28:37.315856 26461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:28:37.326965 26461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:28:37.329051 26461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:28:37.329226 26461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:28:37.333318 26461 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.190:44963
I20260928 06:28:37.333382 26528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.190:44963 every 8 connection(s)
I20260928 06:28:37.333730 26461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
I20260928 06:28:37.334841 26529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:37.337698 26529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57c34d143d78467e8579f5efa006e79d: Bootstrap starting.
I20260928 06:28:37.338356 26529 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57c34d143d78467e8579f5efa006e79d: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:37.338653 26529 log.cc:824] T 00000000000000000000000000000000 P 57c34d143d78467e8579f5efa006e79d: Log is configured to *not* fsync() on all Append() calls
I20260928 06:28:37.339437 26529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57c34d143d78467e8579f5efa006e79d: No bootstrap required, opened a new log
I20260928 06:28:37.340333 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 26461
I20260928 06:28:37.340451 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/minicluster-data/master-0/wal/instance
I20260928 06:28:37.340695 23430 external_mini_cluster.cc:1428] Setting key 4ffa31d56049b8cf2fc6070f31f9d546
I20260928 06:28:37.341885 26529 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57c34d143d78467e8579f5efa006e79d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57c34d143d78467e8579f5efa006e79d" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 44963 } }
I20260928 06:28:37.342123 26529 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57c34d143d78467e8579f5efa006e79d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:37.342196 26529 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57c34d143d78467e8579f5efa006e79d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57c34d143d78467e8579f5efa006e79d, State: Initialized, Role: FOLLOWER
I20260928 06:28:37.342341 26529 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57c34d143d78467e8579f5efa006e79d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57c34d143d78467e8579f5efa006e79d" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 44963 } }
I20260928 06:28:37.342420 26529 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57c34d143d78467e8579f5efa006e79d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:28:37.342449 26529 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57c34d143d78467e8579f5efa006e79d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:28:37.342530 26529 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57c34d143d78467e8579f5efa006e79d [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:37.343588 26529 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57c34d143d78467e8579f5efa006e79d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57c34d143d78467e8579f5efa006e79d" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 44963 } }
I20260928 06:28:37.343698 26529 leader_election.cc:302] T 00000000000000000000000000000000 P 57c34d143d78467e8579f5efa006e79d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57c34d143d78467e8579f5efa006e79d; no voters:
I20260928 06:28:37.343889 26529 leader_election.cc:288] T 00000000000000000000000000000000 P 57c34d143d78467e8579f5efa006e79d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:28:37.343946 26534 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57c34d143d78467e8579f5efa006e79d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:28:37.344144 26534 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57c34d143d78467e8579f5efa006e79d [term 1 LEADER]: Becoming Leader. State: Replica: 57c34d143d78467e8579f5efa006e79d, State: Running, Role: LEADER
I20260928 06:28:37.344177 26529 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57c34d143d78467e8579f5efa006e79d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:28:37.344282 26534 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57c34d143d78467e8579f5efa006e79d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57c34d143d78467e8579f5efa006e79d" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 44963 } }
I20260928 06:28:37.344693 26534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57c34d143d78467e8579f5efa006e79d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57c34d143d78467e8579f5efa006e79d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57c34d143d78467e8579f5efa006e79d" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 44963 } } }
I20260928 06:28:37.344818 26534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57c34d143d78467e8579f5efa006e79d [sys.catalog]: This master's current role is: LEADER
I20260928 06:28:37.345017 26534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57c34d143d78467e8579f5efa006e79d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57c34d143d78467e8579f5efa006e79d. Latest consensus state: current_term: 1 leader_uuid: "57c34d143d78467e8579f5efa006e79d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57c34d143d78467e8579f5efa006e79d" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 44963 } } }
I20260928 06:28:37.345124 26534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57c34d143d78467e8579f5efa006e79d [sys.catalog]: This master's current role is: LEADER
I20260928 06:28:37.345275 26543 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:28:37.345813 26543 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 06:28:37 dist-test-slave-d59m krb5kdc[26448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:28:37.349828 26543 catalog_manager.cc:1347] Generated new cluster ID: b8b9f9d223844aa49d6759d5f1d2cc7d
I20260928 06:28:37.349877 26543 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:28:37.352385 26533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:37.341251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50952 (local address 127.22.225.190:44963)
0928 06:28:37.341439 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:37.341448 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:37.341480 (+ 32us) server_negotiation.cc:407] Connection header received
0928 06:28:37.341856 (+ 376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:37.341863 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:37.341908 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:37.342071 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:37.342842 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:37.343789 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:37.344913 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:37.345549 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:37.345635 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:37.348748 (+ 3113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:37.348752 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:37.348753 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:37.348764 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:37.350654 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:37.351005 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:37.351008 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:37.351008 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:37.351121 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:37.351285 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:37.351286 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:37.351286 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:37.351572 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:37.351698 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:37.352078 (+ 380us) server_negotiation.cc:299] Negotiation successful
0928 06:28:37.352199 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":92,"threads_started":1}
I20260928 06:28:37.361601 26543 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:28:37.362119 26543 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:28:37.366144 26543 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57c34d143d78467e8579f5efa006e79d: Generated new TSK 0
I20260928 06:28:37.366324 26543 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.225.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab.
Entry for principal kudu/127.22.225.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab.
I20260928 06:28:37.399720 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.129:0
--local_ip_for_outbound_sockets=127.22.225.129
--webserver_interface=127.22.225.129
--webserver_port=0
--tserver_master_addrs=127.22.225.190:44963
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab
--principal=kudu/127.22.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:38425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:28:37.480480 26555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:28:37.480770 26555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:28:37.480813 26555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:28:37.482265 26555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:28:37.482350 26555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:28:37.482398 26555 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.129
I20260928 06:28:37.483996 26555 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:38425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab
--principal=kudu/127.22.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.225.129
--webserver_port=0
--tserver_master_addrs=127.22.225.190:44963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26555
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.129
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:28:37.484268 26555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:28:37.484514 26555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:28:37.489444 26561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:37.489511 26560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:37.489848 26563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:37.489956 26555 server_base.cc:1034] running on GCE node
I20260928 06:28:37.490198 26555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:28:37.490461 26555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:28:37.491643 26555 hybrid_clock.cc:648] HybridClock initialized: now 1790576917491621 us; error 36 us; skew 500 ppm
Sep 28 06:28:37 dist-test-slave-d59m krb5kdc[26448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576917, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:28:37.494563 26555 init.cc:377] Logged in from keytab as kudu/127.22.225.129@KRBTEST.COM (short username kudu)
I20260928 06:28:37.495013 26555 webserver.cc:460] Webserver started at http://127.22.225.129:33849/ using document root <none> and password file <none>
I20260928 06:28:37.495275 26555 fs_manager.cc:359] Metadata directory not provided
I20260928 06:28:37.495340 26555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:28:37.495472 26555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:28:37.496484 26555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-0/data/instance:
uuid: "2c5ca53da94449ebaf91dd244d07b3a4"
format_stamp: "Formatted at 2026-09-28 06:28:37 on dist-test-slave-d59m"
server_key: "44717eea01bf25b99e059b8df605d1e3"
server_key_iv: "575e7daf1476274636c0b095f323941e"
server_key_version: "encryptionkey@0"
I20260928 06:28:37.496919 26555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-0/wal/instance:
uuid: "2c5ca53da94449ebaf91dd244d07b3a4"
format_stamp: "Formatted at 2026-09-28 06:28:37 on dist-test-slave-d59m"
server_key: "44717eea01bf25b99e059b8df605d1e3"
server_key_iv: "575e7daf1476274636c0b095f323941e"
server_key_version: "encryptionkey@0"
I20260928 06:28:37.498240 26555 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:28:37.499219 26570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:37.499367 26555 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:28:37.499452 26555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-0/wal
uuid: "2c5ca53da94449ebaf91dd244d07b3a4"
format_stamp: "Formatted at 2026-09-28 06:28:37 on dist-test-slave-d59m"
server_key: "44717eea01bf25b99e059b8df605d1e3"
server_key_iv: "575e7daf1476274636c0b095f323941e"
server_key_version: "encryptionkey@0"
I20260928 06:28:37.499554 26555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:28:37.517273 26555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:28:37.519580 26555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:28:37.519819 26555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:28:37.520061 26555 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:28:37.520367 26555 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:28:37.520421 26555 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:37.520462 26555 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:28:37.520489 26555 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:37.526767 26555 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.129:40035
I20260928 06:28:37.526815 26683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.129:40035 every 8 connection(s)
I20260928 06:28:37.527135 26555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
Sep 28 06:28:37 dist-test-slave-d59m krb5kdc[26448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576917, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.129@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:28:37.534538 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 26555
I20260928 06:28:37.534641 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-0/wal/instance
I20260928 06:28:37.534865 23430 external_mini_cluster.cc:1428] Setting key 6e5b54c02b950f93b42fb1a7dc2ffbc9
I20260928 06:28:37.537057 26533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:37.528111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:40161 (local address 127.22.225.190:44963)
0928 06:28:37.528192 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:37.528194 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:37.528210 (+ 16us) server_negotiation.cc:407] Connection header received
0928 06:28:37.528738 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:37.528739 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:37.528753 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:37.528832 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:37.529721 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:37.530395 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:37.531339 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:37.531610 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:37.531680 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:37.533875 (+ 2195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:37.533876 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:37.533876 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:37.533884 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:37.535452 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:37.535927 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:37.535928 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:37.535928 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:37.535988 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:37.536294 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:37.536295 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:37.536295 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:37.536420 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:37.536488 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:37.536890 (+ 402us) server_negotiation.cc:299] Negotiation successful
0928 06:28:37.536992 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 06:28:37.537544 26686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:37.527838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:44963 (local address 127.22.225.129:40161)
0928 06:28:37.528017 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:37.528059 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:37.528558 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:37.528885 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:37.528888 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:37.529435 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:37.529624 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:37.529628 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:37.530483 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:37.530483 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:37.531251 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:37.531253 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:37.531691 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:37.531691 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:37.531785 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:37.532247 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:37.532255 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:37.533754 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:37.535526 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:37.535532 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:37.535534 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:37.535846 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:37.536062 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:37.536063 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:37.536064 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:37.536206 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:37.536492 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:37.536498 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:37.536710 (+ 212us) client_negotiation.cc:769] Sending connection context
0928 06:28:37.537206 (+ 496us) client_negotiation.cc:240] Negotiation successful
0928 06:28:37.537386 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":107,"threads_started":1}
I20260928 06:28:37.538028 26684 heartbeater.cc:347] Connected to a master server at 127.22.225.190:44963
I20260928 06:28:37.538122 26684 heartbeater.cc:464] Registering TS with master...
I20260928 06:28:37.538309 26684 heartbeater.cc:511] Master 127.22.225.190:44963 requested a full tablet report, sending...
I20260928 06:28:37.538874 26493 ts_manager.cc:194] Registered new tserver with Master: 2c5ca53da94449ebaf91dd244d07b3a4 (127.22.225.129:40035)
I20260928 06:28:37.539615 26493 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.225.129@KRBTEST.COM'} at 127.22.225.129:40161
WARNING: no policy specified for kudu/127.22.225.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab.
Entry for principal kudu/127.22.225.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab.
I20260928 06:28:37.560751 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.130:0
--local_ip_for_outbound_sockets=127.22.225.130
--webserver_interface=127.22.225.130
--webserver_port=0
--tserver_master_addrs=127.22.225.190:44963
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab
--principal=kudu/127.22.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:38425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:28:37.646983 26689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:28:37.647185 26689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:28:37.647208 26689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:28:37.648334 26689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:28:37.648381 26689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:28:37.648408 26689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.130
I20260928 06:28:37.649884 26689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:38425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab
--principal=kudu/127.22.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.225.130
--webserver_port=0
--tserver_master_addrs=127.22.225.190:44963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.130
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:28:37.650117 26689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:28:37.650367 26689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:28:37.655318 26695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:37.655309 26694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:37.655332 26697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:37.655884 26689 server_base.cc:1034] running on GCE node
I20260928 06:28:37.656096 26689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:28:37.656358 26689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:28:37.657532 26689 hybrid_clock.cc:648] HybridClock initialized: now 1790576917657495 us; error 55 us; skew 500 ppm
Sep 28 06:28:37 dist-test-slave-d59m krb5kdc[26448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576917, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:28:37.660663 26689 init.cc:377] Logged in from keytab as kudu/127.22.225.130@KRBTEST.COM (short username kudu)
I20260928 06:28:37.661115 26689 webserver.cc:460] Webserver started at http://127.22.225.130:33989/ using document root <none> and password file <none>
I20260928 06:28:37.661348 26689 fs_manager.cc:359] Metadata directory not provided
I20260928 06:28:37.661442 26689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:28:37.661580 26689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:28:37.662640 26689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-1/data/instance:
uuid: "eed0f0a9e05948279a1c7fb2ae813d3f"
format_stamp: "Formatted at 2026-09-28 06:28:37 on dist-test-slave-d59m"
server_key: "d58e833ef4d89e7f9638ed6c4222bb86"
server_key_iv: "968e221837b8bb8248cf8d63a8a8a35c"
server_key_version: "encryptionkey@0"
I20260928 06:28:37.663046 26689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-1/wal/instance:
uuid: "eed0f0a9e05948279a1c7fb2ae813d3f"
format_stamp: "Formatted at 2026-09-28 06:28:37 on dist-test-slave-d59m"
server_key: "d58e833ef4d89e7f9638ed6c4222bb86"
server_key_iv: "968e221837b8bb8248cf8d63a8a8a35c"
server_key_version: "encryptionkey@0"
I20260928 06:28:37.664433 26689 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:28:37.665365 26704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:37.665552 26689 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:28:37.665647 26689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-1/wal
uuid: "eed0f0a9e05948279a1c7fb2ae813d3f"
format_stamp: "Formatted at 2026-09-28 06:28:37 on dist-test-slave-d59m"
server_key: "d58e833ef4d89e7f9638ed6c4222bb86"
server_key_iv: "968e221837b8bb8248cf8d63a8a8a35c"
server_key_version: "encryptionkey@0"
I20260928 06:28:37.665735 26689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:28:37.675416 26689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:28:37.677783 26689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:28:37.677953 26689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:28:37.678198 26689 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:28:37.678509 26689 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:28:37.678563 26689 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:37.678606 26689 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:28:37.678633 26689 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:37.684686 26689 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.130:45031
I20260928 06:28:37.684739 26817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.130:45031 every 8 connection(s)
I20260928 06:28:37.685081 26689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
I20260928 06:28:37.685624 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 26689
I20260928 06:28:37.685742 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-1/wal/instance
I20260928 06:28:37.685978 23430 external_mini_cluster.cc:1428] Setting key ffa4a914def2b455bc12c746680891ac
Sep 28 06:28:37 dist-test-slave-d59m krb5kdc[26448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576917, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.130@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.225.131@KRBTEST.COM; defaulting to no policy
I20260928 06:28:37.695685 26533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:37.686147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:40611 (local address 127.22.225.190:44963)
0928 06:28:37.686192 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:37.686194 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:37.686205 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:28:37.686825 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:37.686827 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:37.686839 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:37.686914 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:37.688230 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:37.688982 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:37.690242 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:37.690434 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:37.690529 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:37.692797 (+ 2268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:37.692799 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:37.692800 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:37.692807 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:37.694238 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:37.694649 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:37.694651 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:37.694651 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:37.694713 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:37.694970 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:37.694971 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:37.694972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:37.695095 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:37.695167 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:37.695510 (+ 343us) server_negotiation.cc:299] Negotiation successful
0928 06:28:37.695593 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 06:28:37.695865 26820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:37.685853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:44963 (local address 127.22.225.130:40611)
0928 06:28:37.686050 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:37.686101 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:37.686706 (+ 605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:37.686973 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:37.686976 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:37.687822 (+ 846us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:37.688112 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:37.688119 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:37.689096 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:37.689098 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:37.690130 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:37.690132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:37.690551 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:37.690552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:37.690689 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:37.691281 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:37.691290 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:37.692704 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:37.694341 (+ 1637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:37.694348 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:37.694350 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:37.694573 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:37.694781 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:37.694782 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:37.694782 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:37.694897 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:37.695169 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:37.695175 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:37.695376 (+ 201us) client_negotiation.cc:769] Sending connection context
0928 06:28:37.695488 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 06:28:37.695638 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":89,"threads_started":1}
I20260928 06:28:37.696466 26818 heartbeater.cc:347] Connected to a master server at 127.22.225.190:44963
I20260928 06:28:37.696579 26818 heartbeater.cc:464] Registering TS with master...
I20260928 06:28:37.696801 26818 heartbeater.cc:511] Master 127.22.225.190:44963 requested a full tablet report, sending...
I20260928 06:28:37.697271 26493 ts_manager.cc:194] Registered new tserver with Master: eed0f0a9e05948279a1c7fb2ae813d3f (127.22.225.130:45031)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.131@KRBTEST.COM" created.
I20260928 06:28:37.697777 26493 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.225.130@KRBTEST.COM'} at 127.22.225.130:40611
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab.
Entry for principal kudu/127.22.225.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab.
I20260928 06:28:37.715180 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.131:0
--local_ip_for_outbound_sockets=127.22.225.131
--webserver_interface=127.22.225.131
--webserver_port=0
--tserver_master_addrs=127.22.225.190:44963
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab
--principal=kudu/127.22.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:38425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:28:37.812649 26823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:28:37.812879 26823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:28:37.812919 26823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:28:37.814224 26823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:28:37.814291 26823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:28:37.814337 26823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.131
I20260928 06:28:37.815788 26823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:38425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab
--principal=kudu/127.22.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.225.131
--webserver_port=0
--tserver_master_addrs=127.22.225.190:44963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.131
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:28:37.816061 26823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:28:37.816289 26823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:28:37.821574 26829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:37.821597 26831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:37.821602 26828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:37.821776 26823 server_base.cc:1034] running on GCE node
I20260928 06:28:37.821998 26823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:28:37.822263 26823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:28:37.823413 26823 hybrid_clock.cc:648] HybridClock initialized: now 1790576917823405 us; error 29 us; skew 500 ppm
Sep 28 06:28:37 dist-test-slave-d59m krb5kdc[26448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576917, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:28:37.831161 26823 init.cc:377] Logged in from keytab as kudu/127.22.225.131@KRBTEST.COM (short username kudu)
I20260928 06:28:37.831683 26823 webserver.cc:460] Webserver started at http://127.22.225.131:42855/ using document root <none> and password file <none>
I20260928 06:28:37.831933 26823 fs_manager.cc:359] Metadata directory not provided
I20260928 06:28:37.831980 26823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:28:37.832106 26823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:28:37.833249 26823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-2/data/instance:
uuid: "9b28b06183d0489dae296595346d7a8d"
format_stamp: "Formatted at 2026-09-28 06:28:37 on dist-test-slave-d59m"
server_key: "b40f596982aea5814fb13a97a5a33ca8"
server_key_iv: "1f017bcc3eac398310d6cc8ef6540dd8"
server_key_version: "encryptionkey@0"
I20260928 06:28:37.833607 26823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-2/wal/instance:
uuid: "9b28b06183d0489dae296595346d7a8d"
format_stamp: "Formatted at 2026-09-28 06:28:37 on dist-test-slave-d59m"
server_key: "b40f596982aea5814fb13a97a5a33ca8"
server_key_iv: "1f017bcc3eac398310d6cc8ef6540dd8"
server_key_version: "encryptionkey@0"
I20260928 06:28:37.834944 26823 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:28:37.835702 26838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:37.835866 26823 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:28:37.835975 26823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-2/wal
uuid: "9b28b06183d0489dae296595346d7a8d"
format_stamp: "Formatted at 2026-09-28 06:28:37 on dist-test-slave-d59m"
server_key: "b40f596982aea5814fb13a97a5a33ca8"
server_key_iv: "1f017bcc3eac398310d6cc8ef6540dd8"
server_key_version: "encryptionkey@0"
I20260928 06:28:37.836043 26823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:28:37.846784 26823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:28:37.849144 26823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:28:37.849306 26823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:28:37.849558 26823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:28:37.849879 26823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:28:37.849907 26823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:37.849928 26823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:28:37.849941 26823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:37.856910 26823 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.131:42639
I20260928 06:28:37.856940 26951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.131:42639 every 8 connection(s)
I20260928 06:28:37.857344 26823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
I20260928 06:28:37.860607 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 26823
I20260928 06:28:37.860699 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576899143097-23430-0/minicluster-data/ts-2/wal/instance
I20260928 06:28:37.860852 23430 external_mini_cluster.cc:1428] Setting key 9e257343a8848fab659b10bd8f891682
Sep 28 06:28:37 dist-test-slave-d59m krb5kdc[26448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576917, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.131@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:28:37.867484 26533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:37.858128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:32841 (local address 127.22.225.190:44963)
0928 06:28:37.858186 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:37.858188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:37.858399 (+ 211us) server_negotiation.cc:407] Connection header received
0928 06:28:37.858956 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:37.858958 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:37.858970 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:37.859047 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:37.859973 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:37.860531 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:37.862090 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:37.862272 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:37.862322 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:37.864551 (+ 2229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:37.864553 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:37.864553 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:37.864561 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:37.866028 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:37.866519 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:37.866520 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:37.866521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:37.866586 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:37.866853 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:37.866854 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:37.866854 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:37.866965 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:37.867037 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:37.867307 (+ 270us) server_negotiation.cc:299] Negotiation successful
0928 06:28:37.867394 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 06:28:37.867712 26954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:37.858120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:44963 (local address 127.22.225.131:32841)
0928 06:28:37.858279 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:37.858313 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:37.858815 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:37.859113 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:37.859117 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:37.859683 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:37.859874 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:37.859880 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:37.860673 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:37.860674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:37.861995 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:37.861998 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:37.862347 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:37.862348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:37.862494 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:37.862966 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:37.862976 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:37.864457 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:37.866122 (+ 1665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:37.866128 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:37.866130 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:37.866426 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:37.866657 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:37.866658 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:37.866659 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:37.866772 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:37.867038 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:37.867044 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:37.867178 (+ 134us) client_negotiation.cc:769] Sending connection context
0928 06:28:37.867285 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 06:28:37.867440 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":75,"threads_started":1}
I20260928 06:28:37.868172 26952 heartbeater.cc:347] Connected to a master server at 127.22.225.190:44963
I20260928 06:28:37.868278 26952 heartbeater.cc:464] Registering TS with master...
I20260928 06:28:37.868470 26952 heartbeater.cc:511] Master 127.22.225.190:44963 requested a full tablet report, sending...
I20260928 06:28:37.868876 26493 ts_manager.cc:194] Registered new tserver with Master: 9b28b06183d0489dae296595346d7a8d (127.22.225.131:42639)
I20260928 06:28:37.869258 26493 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.225.131@KRBTEST.COM'} at 127.22.225.131:32841
I20260928 06:28:37.872345 23430 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 06:28:37 dist-test-slave-d59m krb5kdc[26448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576917, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 06:28:37 dist-test-slave-d59m krb5kdc[26448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576917, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:28:37.896883 26533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:37.889434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50962 (local address 127.22.225.190:44963)
0928 06:28:37.889494 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:37.889496 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:37.889601 (+ 105us) server_negotiation.cc:407] Connection header received
0928 06:28:37.889649 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:37.889650 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:37.889661 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:37.889771 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:37.890340 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:37.890830 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:37.891614 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:37.891803 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:37.891907 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:37.894146 (+ 2239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:37.894147 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:37.894148 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:37.894156 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:37.895662 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:37.896016 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:37.896017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:37.896017 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:37.896074 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:37.896289 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:37.896290 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:37.896290 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:37.896458 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:37.896568 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:37.896701 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 06:28:37.896778 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260928 06:28:37.897135 26493 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:50962
I20260928 06:28:37.897629 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 26555
I20260928 06:28:37.903692 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 26689
I20260928 06:28:37.908993 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 26823
I20260928 06:28:37.913473 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 26461
2026-09-28T06:28:37Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1944 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:28:37 dist-test-slave-d59m krb5kdc[26964](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:28:37 dist-test-slave-d59m krb5kdc[26964](info): set up 2 sockets
Sep 28 06:28:37 dist-test-slave-d59m krb5kdc[26964](info): commencing operation
krb5kdc: starting...
W20260928 06:28:38.981267 23430 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Sep 28 06:28:39 dist-test-slave-d59m krb5kdc[26964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:28:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:28:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab.
Entry for principal kudu/127.22.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab.
I20260928 06:28:39.103672 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:42147
--webserver_interface=127.22.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab
--principal=kudu/127.22.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:43223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.225.190:42147
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:28:39.217478 26977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:28:39.217689 26977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:28:39.217710 26977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:28:39.218979 26977 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:28:39.219022 26977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:28:39.219036 26977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:28:39.219049 26977 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:28:39.219061 26977 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:28:39.220681 26977 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:43223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.225.190:42147
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab
--principal=kudu/127.22.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:42147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:28:39.220917 26977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:28:39.221151 26977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:28:39.227195 26983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:39.227216 26982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:39.227562 26977 server_base.cc:1034] running on GCE node
W20260928 06:28:39.227216 26985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:39.227887 26977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:28:39.228186 26977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:28:39.229357 26977 hybrid_clock.cc:648] HybridClock initialized: now 1790576919229332 us; error 44 us; skew 500 ppm
Sep 28 06:28:39 dist-test-slave-d59m krb5kdc[26964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576919, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:28:39.232115 26977 init.cc:377] Logged in from keytab as kudu/127.22.225.190@KRBTEST.COM (short username kudu)
I20260928 06:28:39.232707 26977 webserver.cc:460] Webserver started at http://127.22.225.190:45331/ using document root <none> and password file <none>
I20260928 06:28:39.232933 26977 fs_manager.cc:359] Metadata directory not provided
I20260928 06:28:39.233002 26977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:28:39.233157 26977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:28:39.234217 26977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/minicluster-data/master-0/data/instance:
uuid: "41d1a63ec0ef4b9b9baa2392686a5635"
format_stamp: "Formatted at 2026-09-28 06:28:39 on dist-test-slave-d59m"
server_key: "f29c16533f780a4eecbe5c9780c36d60"
server_key_iv: "635620f35393c2f69d8796676de8ffbd"
server_key_version: "encryptionkey@0"
I20260928 06:28:39.234593 26977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/minicluster-data/master-0/wal/instance:
uuid: "41d1a63ec0ef4b9b9baa2392686a5635"
format_stamp: "Formatted at 2026-09-28 06:28:39 on dist-test-slave-d59m"
server_key: "f29c16533f780a4eecbe5c9780c36d60"
server_key_iv: "635620f35393c2f69d8796676de8ffbd"
server_key_version: "encryptionkey@0"
I20260928 06:28:39.235965 26977 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:28:39.236824 26992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:39.237041 26977 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:28:39.237146 26977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/minicluster-data/master-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/minicluster-data/master-0/wal
uuid: "41d1a63ec0ef4b9b9baa2392686a5635"
format_stamp: "Formatted at 2026-09-28 06:28:39 on dist-test-slave-d59m"
server_key: "f29c16533f780a4eecbe5c9780c36d60"
server_key_iv: "635620f35393c2f69d8796676de8ffbd"
server_key_version: "encryptionkey@0"
I20260928 06:28:39.237227 26977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:28:39.272181 26977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:28:39.274489 26977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:28:39.274638 26977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:28:39.279560 26977 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.190:42147
I20260928 06:28:39.279592 27044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.190:42147 every 8 connection(s)
I20260928 06:28:39.280112 26977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
I20260928 06:28:39.280963 27045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:39.284198 27045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41d1a63ec0ef4b9b9baa2392686a5635: Bootstrap starting.
I20260928 06:28:39.284960 27045 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41d1a63ec0ef4b9b9baa2392686a5635: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:39.285310 27045 log.cc:824] T 00000000000000000000000000000000 P 41d1a63ec0ef4b9b9baa2392686a5635: Log is configured to *not* fsync() on all Append() calls
I20260928 06:28:39.286161 27045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41d1a63ec0ef4b9b9baa2392686a5635: No bootstrap required, opened a new log
I20260928 06:28:39.288374 27045 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41d1a63ec0ef4b9b9baa2392686a5635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41d1a63ec0ef4b9b9baa2392686a5635" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 42147 } }
I20260928 06:28:39.288560 27045 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41d1a63ec0ef4b9b9baa2392686a5635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:39.288640 27045 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41d1a63ec0ef4b9b9baa2392686a5635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41d1a63ec0ef4b9b9baa2392686a5635, State: Initialized, Role: FOLLOWER
I20260928 06:28:39.288796 27045 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41d1a63ec0ef4b9b9baa2392686a5635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41d1a63ec0ef4b9b9baa2392686a5635" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 42147 } }
I20260928 06:28:39.288880 27045 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41d1a63ec0ef4b9b9baa2392686a5635 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:28:39.288913 27045 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41d1a63ec0ef4b9b9baa2392686a5635 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:28:39.288961 27045 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41d1a63ec0ef4b9b9baa2392686a5635 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:39.289975 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 26977
I20260928 06:28:39.290081 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/minicluster-data/master-0/wal/instance
I20260928 06:28:39.290199 27045 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41d1a63ec0ef4b9b9baa2392686a5635 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41d1a63ec0ef4b9b9baa2392686a5635" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 42147 } }
I20260928 06:28:39.290306 23430 external_mini_cluster.cc:1428] Setting key d8b63c7915522064c69476bdaae9474a
I20260928 06:28:39.290310 27045 leader_election.cc:302] T 00000000000000000000000000000000 P 41d1a63ec0ef4b9b9baa2392686a5635 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41d1a63ec0ef4b9b9baa2392686a5635; no voters:
I20260928 06:28:39.290539 27045 leader_election.cc:288] T 00000000000000000000000000000000 P 41d1a63ec0ef4b9b9baa2392686a5635 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:28:39.290577 27048 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41d1a63ec0ef4b9b9baa2392686a5635 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:28:39.290827 27048 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41d1a63ec0ef4b9b9baa2392686a5635 [term 1 LEADER]: Becoming Leader. State: Replica: 41d1a63ec0ef4b9b9baa2392686a5635, State: Running, Role: LEADER
I20260928 06:28:39.291018 27048 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41d1a63ec0ef4b9b9baa2392686a5635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41d1a63ec0ef4b9b9baa2392686a5635" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 42147 } }
I20260928 06:28:39.290877 27045 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41d1a63ec0ef4b9b9baa2392686a5635 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:28:39.291421 27048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41d1a63ec0ef4b9b9baa2392686a5635 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41d1a63ec0ef4b9b9baa2392686a5635" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41d1a63ec0ef4b9b9baa2392686a5635" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 42147 } } }
I20260928 06:28:39.291575 27048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41d1a63ec0ef4b9b9baa2392686a5635 [sys.catalog]: This master's current role is: LEADER
I20260928 06:28:39.291765 27048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41d1a63ec0ef4b9b9baa2392686a5635 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41d1a63ec0ef4b9b9baa2392686a5635. Latest consensus state: current_term: 1 leader_uuid: "41d1a63ec0ef4b9b9baa2392686a5635" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41d1a63ec0ef4b9b9baa2392686a5635" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 42147 } } }
I20260928 06:28:39.291885 27048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41d1a63ec0ef4b9b9baa2392686a5635 [sys.catalog]: This master's current role is: LEADER
I20260928 06:28:39.292052 27057 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:28:39.292618 27057 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:28:39.294867 27067 mvcc.cc:205] Tried to move back new op lower bound from 7334203061426601984 to 7334203061416726528. Current Snapshot: MvccSnapshot[applied={T|T < 7334203061426601984}]
I20260928 06:28:39.295342 27057 catalog_manager.cc:1347] Generated new cluster ID: 36c38c2beae646ccac4052ff3c2a8e60
I20260928 06:28:39.295399 27057 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:28:39 dist-test-slave-d59m krb5kdc[26964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:28:39.302611 27052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:39.291013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50142 (local address 127.22.225.190:42147)
0928 06:28:39.291276 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:39.291285 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:39.291322 (+ 37us) server_negotiation.cc:407] Connection header received
0928 06:28:39.291442 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:39.291449 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:39.291507 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:39.292430 (+ 923us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:39.292531 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:39.294074 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:39.295115 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:39.295698 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:39.295803 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:39.298596 (+ 2793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:39.298601 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:39.298601 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:39.298613 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:39.300563 (+ 1950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:39.300958 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:39.300961 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:39.300962 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:39.301104 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:39.301311 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:39.301312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:39.301312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:39.301596 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:39.301767 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:39.302278 (+ 511us) server_negotiation.cc:299] Negotiation successful
0928 06:28:39.302412 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":71,"threads_started":1}
I20260928 06:28:39.341419 27057 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:28:39.342190 27057 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:28:39.349714 27057 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41d1a63ec0ef4b9b9baa2392686a5635: Generated new TSK 0
I20260928 06:28:39.349892 27057 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.225.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab.
Entry for principal kudu/127.22.225.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab.
I20260928 06:28:39.380067 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.129:0
--local_ip_for_outbound_sockets=127.22.225.129
--webserver_interface=127.22.225.129
--webserver_port=0
--tserver_master_addrs=127.22.225.190:42147
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab
--principal=kudu/127.22.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:43223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:28:39.490813 27071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:28:39.491008 27071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:28:39.491031 27071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:28:39.492275 27071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:28:39.492317 27071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:28:39.492343 27071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.129
I20260928 06:28:39.493899 27071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:43223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab
--principal=kudu/127.22.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.225.129
--webserver_port=0
--tserver_master_addrs=127.22.225.190:42147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.129
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:28:39.494133 27071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:28:39.494377 27071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:28:39.499490 27076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:39.499533 27077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:39.499791 27071 server_base.cc:1034] running on GCE node
W20260928 06:28:39.499507 27079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:39.500108 27071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:28:39.500397 27071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:28:39.501566 27071 hybrid_clock.cc:648] HybridClock initialized: now 1790576919501536 us; error 50 us; skew 500 ppm
Sep 28 06:28:39 dist-test-slave-d59m krb5kdc[26964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576919, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:28:39.504078 27071 init.cc:377] Logged in from keytab as kudu/127.22.225.129@KRBTEST.COM (short username kudu)
I20260928 06:28:39.504640 27071 webserver.cc:460] Webserver started at http://127.22.225.129:41459/ using document root <none> and password file <none>
I20260928 06:28:39.504884 27071 fs_manager.cc:359] Metadata directory not provided
I20260928 06:28:39.504928 27071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:28:39.505070 27071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:28:39.506213 27071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/minicluster-data/ts-0/data/instance:
uuid: "6fb5c7cad938404e9912f4e65699dafd"
format_stamp: "Formatted at 2026-09-28 06:28:39 on dist-test-slave-d59m"
server_key: "2c870a3e0c04332027c42f6c9e8b1b6b"
server_key_iv: "646d3cd26c1193f8a592ae91c513dc5e"
server_key_version: "encryptionkey@0"
I20260928 06:28:39.506588 27071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/minicluster-data/ts-0/wal/instance:
uuid: "6fb5c7cad938404e9912f4e65699dafd"
format_stamp: "Formatted at 2026-09-28 06:28:39 on dist-test-slave-d59m"
server_key: "2c870a3e0c04332027c42f6c9e8b1b6b"
server_key_iv: "646d3cd26c1193f8a592ae91c513dc5e"
server_key_version: "encryptionkey@0"
I20260928 06:28:39.508001 27071 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:28:39.508836 27086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:39.509028 27071 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:28:39.509125 27071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/minicluster-data/ts-0/wal
uuid: "6fb5c7cad938404e9912f4e65699dafd"
format_stamp: "Formatted at 2026-09-28 06:28:39 on dist-test-slave-d59m"
server_key: "2c870a3e0c04332027c42f6c9e8b1b6b"
server_key_iv: "646d3cd26c1193f8a592ae91c513dc5e"
server_key_version: "encryptionkey@0"
I20260928 06:28:39.509218 27071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:28:39.532303 27071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:28:39.534574 27071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:28:39.534740 27071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:28:39.534993 27071 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:28:39.535331 27071 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:28:39.535386 27071 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:39.535427 27071 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:28:39.535454 27071 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:39.541661 27071 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.129:44411
I20260928 06:28:39.541710 27199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.129:44411 every 8 connection(s)
I20260928 06:28:39.542032 27071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
I20260928 06:28:39.545939 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 27071
I20260928 06:28:39.546054 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/minicluster-data/ts-0/wal/instance
I20260928 06:28:39.546236 23430 external_mini_cluster.cc:1428] Setting key 06ad2014262e190a0dee0546b4a13141
Sep 28 06:28:39 dist-test-slave-d59m krb5kdc[26964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576919, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.129@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:28:39.552695 27052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:39.542822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:40543 (local address 127.22.225.190:42147)
0928 06:28:39.542893 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:39.542896 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:39.543094 (+ 198us) server_negotiation.cc:407] Connection header received
0928 06:28:39.543991 (+ 897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:39.543993 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:39.544016 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:39.544097 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:39.544767 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:39.545554 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:39.546964 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:39.547188 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:39.547264 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:39.549782 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:39.549784 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:39.549784 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:39.549795 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:39.551259 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:39.551692 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:39.551693 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:39.551694 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:39.551753 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:39.552002 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:39.552003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:39.552004 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:39.552121 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:39.552194 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:39.552491 (+ 297us) server_negotiation.cc:299] Negotiation successful
0928 06:28:39.552574 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 06:28:39.552948 27202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:39.542781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:42147 (local address 127.22.225.129:40543)
0928 06:28:39.542967 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:39.543006 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:39.543888 (+ 882us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:39.544166 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:39.544169 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:39.544407 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:39.544669 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:39.544675 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:39.545643 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:39.545644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:39.546869 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:39.546872 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:39.547271 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:39.547291 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:39.547420 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:39.547951 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:39.547958 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:39.549677 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:39.551362 (+ 1685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:39.551369 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:39.551371 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:39.551615 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:39.551829 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:39.551830 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:39.551830 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:39.551937 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:39.552196 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:39.552204 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:39.552338 (+ 134us) client_negotiation.cc:769] Sending connection context
0928 06:28:39.552455 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 06:28:39.552644 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":116,"threads_started":1}
I20260928 06:28:39.553507 27200 heartbeater.cc:347] Connected to a master server at 127.22.225.190:42147
I20260928 06:28:39.553604 27200 heartbeater.cc:464] Registering TS with master...
I20260928 06:28:39.553788 27200 heartbeater.cc:511] Master 127.22.225.190:42147 requested a full tablet report, sending...
I20260928 06:28:39.554315 27009 ts_manager.cc:194] Registered new tserver with Master: 6fb5c7cad938404e9912f4e65699dafd (127.22.225.129:44411)
I20260928 06:28:39.554919 27009 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.225.129@KRBTEST.COM'} at 127.22.225.129:40543
WARNING: no policy specified for kudu/127.22.225.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab.
Entry for principal kudu/127.22.225.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab.
I20260928 06:28:39.573081 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.130:0
--local_ip_for_outbound_sockets=127.22.225.130
--webserver_interface=127.22.225.130
--webserver_port=0
--tserver_master_addrs=127.22.225.190:42147
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab
--principal=kudu/127.22.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:43223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:28:39.678125 27205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:28:39.678359 27205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:28:39.678400 27205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:28:39.679618 27205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:28:39.679684 27205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:28:39.679730 27205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.130
I20260928 06:28:39.681825 27205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:43223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab
--principal=kudu/127.22.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.225.130
--webserver_port=0
--tserver_master_addrs=127.22.225.190:42147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.130
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:28:39.682097 27205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:28:39.682329 27205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:28:39.687459 27213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:39.687510 27211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:39.687512 27210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:39.687963 27205 server_base.cc:1034] running on GCE node
I20260928 06:28:39.688185 27205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:28:39.688431 27205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:28:39.689596 27205 hybrid_clock.cc:648] HybridClock initialized: now 1790576919689572 us; error 44 us; skew 500 ppm
Sep 28 06:28:39 dist-test-slave-d59m krb5kdc[26964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576919, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:28:39.692396 27205 init.cc:377] Logged in from keytab as kudu/127.22.225.130@KRBTEST.COM (short username kudu)
I20260928 06:28:39.692917 27205 webserver.cc:460] Webserver started at http://127.22.225.130:41665/ using document root <none> and password file <none>
I20260928 06:28:39.693161 27205 fs_manager.cc:359] Metadata directory not provided
I20260928 06:28:39.693231 27205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:28:39.693379 27205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:28:39.694439 27205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/minicluster-data/ts-1/data/instance:
uuid: "85807c52e61d4308b54b8e114cd0115f"
format_stamp: "Formatted at 2026-09-28 06:28:39 on dist-test-slave-d59m"
server_key: "94e496eacb0bb96f1b359ade99fe7566"
server_key_iv: "4f07446b2b6c405d036248e405a2ad67"
server_key_version: "encryptionkey@0"
I20260928 06:28:39.694820 27205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/minicluster-data/ts-1/wal/instance:
uuid: "85807c52e61d4308b54b8e114cd0115f"
format_stamp: "Formatted at 2026-09-28 06:28:39 on dist-test-slave-d59m"
server_key: "94e496eacb0bb96f1b359ade99fe7566"
server_key_iv: "4f07446b2b6c405d036248e405a2ad67"
server_key_version: "encryptionkey@0"
I20260928 06:28:39.696203 27205 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:28:39.697044 27220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:39.697220 27205 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:28:39.697309 27205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/minicluster-data/ts-1/wal
uuid: "85807c52e61d4308b54b8e114cd0115f"
format_stamp: "Formatted at 2026-09-28 06:28:39 on dist-test-slave-d59m"
server_key: "94e496eacb0bb96f1b359ade99fe7566"
server_key_iv: "4f07446b2b6c405d036248e405a2ad67"
server_key_version: "encryptionkey@0"
I20260928 06:28:39.697395 27205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:28:39.716368 27205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:28:39.719108 27205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:28:39.719257 27205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:28:39.719497 27205 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:28:39.719852 27205 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:28:39.719897 27205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:39.719928 27205 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:28:39.719954 27205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:39.727278 27205 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.130:37263
I20260928 06:28:39.727345 27333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.130:37263 every 8 connection(s)
I20260928 06:28:39.727737 27205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
I20260928 06:28:39.728467 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 27205
I20260928 06:28:39.728570 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/minicluster-data/ts-1/wal/instance
I20260928 06:28:39.728799 23430 external_mini_cluster.cc:1428] Setting key becebcc0e1219345311fb0f4b3d45f4c
Sep 28 06:28:39 dist-test-slave-d59m krb5kdc[26964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576919, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.130@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.225.131@KRBTEST.COM; defaulting to no policy
I20260928 06:28:39.739391 27052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:39.728558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:48193 (local address 127.22.225.190:42147)
0928 06:28:39.728823 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:39.728826 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:39.728908 (+ 82us) server_negotiation.cc:407] Connection header received
0928 06:28:39.730170 (+ 1262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:39.730172 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:39.730184 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:39.730251 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:39.731066 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:39.731855 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:39.732927 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:39.733192 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:39.733281 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:39.736026 (+ 2745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:39.736028 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:39.736029 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:39.736037 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:39.737847 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:39.738360 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:39.738362 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:39.738362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:39.738415 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:39.738699 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:39.738700 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:39.738700 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:39.738800 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:39.738852 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:39.739235 (+ 383us) server_negotiation.cc:299] Negotiation successful
0928 06:28:39.739323 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232}
I20260928 06:28:39.739650 27336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:39.728536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:42147 (local address 127.22.225.130:48193)
0928 06:28:39.728759 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:39.728810 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:39.730048 (+ 1238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:39.730324 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:39.730329 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:39.730687 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:39.730944 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:39.730950 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:39.731956 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:39.731957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:39.732833 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:39.732835 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:39.733292 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:39.733293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:39.733427 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:39.734025 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:39.734036 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:39.735914 (+ 1878us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:39.737939 (+ 2025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:39.737947 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:39.737949 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:39.738268 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:39.738488 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:39.738489 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:39.738490 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:39.738631 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:39.738867 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:39.738872 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:39.739042 (+ 170us) client_negotiation.cc:769] Sending connection context
0928 06:28:39.739192 (+ 150us) client_negotiation.cc:240] Negotiation successful
0928 06:28:39.739363 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":110,"threads_started":1}
I20260928 06:28:39.740209 27334 heartbeater.cc:347] Connected to a master server at 127.22.225.190:42147
I20260928 06:28:39.740307 27334 heartbeater.cc:464] Registering TS with master...
I20260928 06:28:39.740535 27334 heartbeater.cc:511] Master 127.22.225.190:42147 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.131@KRBTEST.COM" created.
I20260928 06:28:39.740989 27009 ts_manager.cc:194] Registered new tserver with Master: 85807c52e61d4308b54b8e114cd0115f (127.22.225.130:37263)
I20260928 06:28:39.741513 27009 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.225.130@KRBTEST.COM'} at 127.22.225.130:48193
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab.
Entry for principal kudu/127.22.225.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab.
I20260928 06:28:39.755759 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.131:0
--local_ip_for_outbound_sockets=127.22.225.131
--webserver_interface=127.22.225.131
--webserver_port=0
--tserver_master_addrs=127.22.225.190:42147
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab
--principal=kudu/127.22.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:43223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:28:39.843349 27339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:28:39.843531 27339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:28:39.843556 27339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:28:39.844732 27339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:28:39.844769 27339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:28:39.844796 27339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.131
I20260928 06:28:39.846302 27339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:43223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab
--principal=kudu/127.22.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.225.131
--webserver_port=0
--tserver_master_addrs=127.22.225.190:42147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27339
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.131
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:28:39.846539 27339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:28:39.846762 27339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:28:39.852648 27347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:39.852662 27345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:39.852768 27339 server_base.cc:1034] running on GCE node
W20260928 06:28:39.852872 27344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:39.853114 27339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:28:39.853374 27339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:28:39.854526 27339 hybrid_clock.cc:648] HybridClock initialized: now 1790576919854501 us; error 44 us; skew 500 ppm
Sep 28 06:28:39 dist-test-slave-d59m krb5kdc[26964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576919, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:28:39.857124 27339 init.cc:377] Logged in from keytab as kudu/127.22.225.131@KRBTEST.COM (short username kudu)
I20260928 06:28:39.857687 27339 webserver.cc:460] Webserver started at http://127.22.225.131:40681/ using document root <none> and password file <none>
I20260928 06:28:39.857939 27339 fs_manager.cc:359] Metadata directory not provided
I20260928 06:28:39.857995 27339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:28:39.858120 27339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:28:39.859400 27339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/minicluster-data/ts-2/data/instance:
uuid: "b17e4897f9114987990a2ebbfeeadeea"
format_stamp: "Formatted at 2026-09-28 06:28:39 on dist-test-slave-d59m"
server_key: "5e4723be92a3e6c08bab02712999bfe7"
server_key_iv: "eb208a38a0131e3a8de36a54d1fe87fa"
server_key_version: "encryptionkey@0"
I20260928 06:28:39.859774 27339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/minicluster-data/ts-2/wal/instance:
uuid: "b17e4897f9114987990a2ebbfeeadeea"
format_stamp: "Formatted at 2026-09-28 06:28:39 on dist-test-slave-d59m"
server_key: "5e4723be92a3e6c08bab02712999bfe7"
server_key_iv: "eb208a38a0131e3a8de36a54d1fe87fa"
server_key_version: "encryptionkey@0"
I20260928 06:28:39.861362 27339 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:28:39.862166 27354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:39.862360 27339 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:28:39.862447 27339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/minicluster-data/ts-2/wal
uuid: "b17e4897f9114987990a2ebbfeeadeea"
format_stamp: "Formatted at 2026-09-28 06:28:39 on dist-test-slave-d59m"
server_key: "5e4723be92a3e6c08bab02712999bfe7"
server_key_iv: "eb208a38a0131e3a8de36a54d1fe87fa"
server_key_version: "encryptionkey@0"
I20260928 06:28:39.862520 27339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:28:39.882067 27339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:28:39.884289 27339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:28:39.884430 27339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:28:39.884722 27339 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:28:39.885035 27339 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:28:39.885090 27339 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:39.885144 27339 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:28:39.885171 27339 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:39.890974 27339 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.131:46713
I20260928 06:28:39.891055 27467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.131:46713 every 8 connection(s)
I20260928 06:28:39.891366 27339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
I20260928 06:28:39.893388 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 27339
I20260928 06:28:39.893497 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576899143097-23430-0/minicluster-data/ts-2/wal/instance
I20260928 06:28:39.893698 23430 external_mini_cluster.cc:1428] Setting key 746d0994b889cceaa181285b03b395cd
Sep 28 06:28:39 dist-test-slave-d59m krb5kdc[26964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576919, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.131@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:28:39.901880 27052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:39.892254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:57387 (local address 127.22.225.190:42147)
0928 06:28:39.892306 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:39.892308 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:39.892565 (+ 257us) server_negotiation.cc:407] Connection header received
0928 06:28:39.893663 (+ 1098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:39.893664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:39.893677 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:39.893733 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:39.894683 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:39.895195 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:39.896326 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:39.896543 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:39.896680 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:39.898949 (+ 2269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:39.898950 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:39.898951 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:39.898959 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:39.900381 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:39.900845 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:39.900847 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:39.900847 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:39.900910 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:39.901197 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:39.901198 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:39.901199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:39.901352 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:39.901426 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:39.901694 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 06:28:39.901770 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 06:28:39.902081 27470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:39.892229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:42147 (local address 127.22.225.131:57387)
0928 06:28:39.892389 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:39.892429 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:39.893549 (+ 1120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:39.893802 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:39.893806 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:39.894300 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:39.894583 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:39.894590 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:39.895300 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:39.895301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:39.896239 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:39.896242 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:39.896688 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:39.896689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:39.896829 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:39.897350 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:39.897358 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:39.898842 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:39.900475 (+ 1633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:39.900481 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:39.900484 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:39.900769 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:39.900977 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:39.900978 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:39.900979 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:39.901136 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:39.901418 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:39.901423 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:39.901564 (+ 141us) client_negotiation.cc:769] Sending connection context
0928 06:28:39.901672 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 06:28:39.901838 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":93,"threads_started":1}
I20260928 06:28:39.902679 27468 heartbeater.cc:347] Connected to a master server at 127.22.225.190:42147
I20260928 06:28:39.902755 27468 heartbeater.cc:464] Registering TS with master...
I20260928 06:28:39.902954 27468 heartbeater.cc:511] Master 127.22.225.190:42147 requested a full tablet report, sending...
I20260928 06:28:39.903339 27009 ts_manager.cc:194] Registered new tserver with Master: b17e4897f9114987990a2ebbfeeadeea (127.22.225.131:46713)
I20260928 06:28:39.903854 27009 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.225.131@KRBTEST.COM'} at 127.22.225.131:57387
I20260928 06:28:39.905314 23430 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 06:28:39 dist-test-slave-d59m krb5kdc[26964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 06:28:39 dist-test-slave-d59m krb5kdc[26964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.225.129@KRBTEST.COM
I20260928 06:28:39.931087 27475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:39.922545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37178 (local address 127.22.225.129:44411)
0928 06:28:39.922736 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:39.922741 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:39.922751 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:28:39.922778 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:39.922780 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:39.922795 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:39.922896 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:39.923480 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:39.924226 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:39.925070 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:39.925397 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:39.925491 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:39.927943 (+ 2452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:39.927950 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:39.927951 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:39.927960 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:39.929878 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:39.930254 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:39.930257 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:39.930257 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:39.930312 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:39.930521 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:39.930522 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:39.930522 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:39.930660 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:39.930824 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:39.930871 (+ 47us) server_negotiation.cc:299] Negotiation successful
0928 06:28:39.930949 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":106,"threads_started":1}
Sep 28 06:28:39 dist-test-slave-d59m krb5kdc[26964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:28:39.939707 27052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:39.932419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50144 (local address 127.22.225.190:42147)
0928 06:28:39.932473 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:39.932475 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:39.932486 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:28:39.932525 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:39.932526 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:39.932535 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:39.932604 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:39.933079 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:39.933555 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:39.934241 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:39.934446 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:39.934533 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:39.936845 (+ 2312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:39.936846 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:39.936847 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:39.936853 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:39.938319 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:39.938767 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:39.938769 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:39.938769 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:39.938830 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:39.939057 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:39.939083 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:39.939083 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:39.939288 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:39.939400 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:39.939515 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 06:28:39.939605 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
W20260928 06:28:39.940032 27009 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50144
I20260928 06:28:39.940424 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 27071
I20260928 06:28:39.946249 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 27205
I20260928 06:28:39.950824 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 27339
I20260928 06:28:39.955514 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 26977
2026-09-28T06:28:39Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2043 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:28:40 dist-test-slave-d59m krb5kdc[27480](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:28:40 dist-test-slave-d59m krb5kdc[27480](info): set up 2 sockets
Sep 28 06:28:40 dist-test-slave-d59m krb5kdc[27480](info): commencing operation
krb5kdc: starting...
W20260928 06:28:41.041163 23430 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Sep 28 06:28:41 dist-test-slave-d59m krb5kdc[27480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:28:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:28:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab.
Entry for principal kudu/127.22.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab.
I20260928 06:28:41.163902 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:42001
--webserver_interface=127.22.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab
--principal=kudu/127.22.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:38693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.225.190:42001
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:28:41.245656 27493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:28:41.245945 27493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:28:41.245985 27493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:28:41.247166 27493 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:28:41.247228 27493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:28:41.247251 27493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:28:41.247272 27493 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:28:41.247299 27493 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:28:41.248884 27493 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:38693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.225.190:42001
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab
--principal=kudu/127.22.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:42001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:28:41.249161 27493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:28:41.249394 27493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:28:41.255255 27501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:41.255241 27499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:41.255442 27498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:41.255613 27493 server_base.cc:1034] running on GCE node
I20260928 06:28:41.255857 27493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:28:41.256176 27493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:28:41.257333 27493 hybrid_clock.cc:648] HybridClock initialized: now 1790576921257314 us; error 40 us; skew 500 ppm
Sep 28 06:28:41 dist-test-slave-d59m krb5kdc[27480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576921, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:28:41.259938 27493 init.cc:377] Logged in from keytab as kudu/127.22.225.190@KRBTEST.COM (short username kudu)
I20260928 06:28:41.260339 27493 fs_manager.cc:359] Metadata directory not provided
I20260928 06:28:41.260408 27493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:28:41.260560 27493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:28:41.261651 27493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/minicluster-data/master-0/data/instance:
uuid: "d71bb86412684ec1a52f31347cd76025"
format_stamp: "Formatted at 2026-09-28 06:28:41 on dist-test-slave-d59m"
server_key: "da75f3545fa072ac5a41e1ca8bb927d1"
server_key_iv: "6c123c3c58a20d1f907ae4d7f2cdc877"
server_key_version: "encryptionkey@0"
I20260928 06:28:41.262022 27493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/minicluster-data/master-0/wal/instance:
uuid: "d71bb86412684ec1a52f31347cd76025"
format_stamp: "Formatted at 2026-09-28 06:28:41 on dist-test-slave-d59m"
server_key: "da75f3545fa072ac5a41e1ca8bb927d1"
server_key_iv: "6c123c3c58a20d1f907ae4d7f2cdc877"
server_key_version: "encryptionkey@0"
I20260928 06:28:41.263373 27493 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:28:41.264144 27507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:41.264325 27493 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:28:41.264423 27493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/minicluster-data/master-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/minicluster-data/master-0/wal
uuid: "d71bb86412684ec1a52f31347cd76025"
format_stamp: "Formatted at 2026-09-28 06:28:41 on dist-test-slave-d59m"
server_key: "da75f3545fa072ac5a41e1ca8bb927d1"
server_key_iv: "6c123c3c58a20d1f907ae4d7f2cdc877"
server_key_version: "encryptionkey@0"
I20260928 06:28:41.264513 27493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:28:41.276292 27493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:28:41.278745 27493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:28:41.278915 27493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:28:41.282923 27493 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.190:42001
I20260928 06:28:41.282959 27559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.190:42001 every 8 connection(s)
I20260928 06:28:41.283257 27493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
I20260928 06:28:41.284340 27560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:41.287421 27560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d71bb86412684ec1a52f31347cd76025: Bootstrap starting.
I20260928 06:28:41.288120 27560 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d71bb86412684ec1a52f31347cd76025: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:41.288427 27560 log.cc:824] T 00000000000000000000000000000000 P d71bb86412684ec1a52f31347cd76025: Log is configured to *not* fsync() on all Append() calls
I20260928 06:28:41.288800 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 27493
I20260928 06:28:41.288920 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/minicluster-data/master-0/wal/instance
I20260928 06:28:41.289112 23430 external_mini_cluster.cc:1428] Setting key f05fd97e758a5886706bcbe0a1930dfb
I20260928 06:28:41.289244 27560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d71bb86412684ec1a52f31347cd76025: No bootstrap required, opened a new log
I20260928 06:28:41.291000 27560 raft_consensus.cc:348] T 00000000000000000000000000000000 P d71bb86412684ec1a52f31347cd76025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d71bb86412684ec1a52f31347cd76025" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 42001 } }
I20260928 06:28:41.291155 27560 raft_consensus.cc:374] T 00000000000000000000000000000000 P d71bb86412684ec1a52f31347cd76025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:41.291219 27560 raft_consensus.cc:729] T 00000000000000000000000000000000 P d71bb86412684ec1a52f31347cd76025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d71bb86412684ec1a52f31347cd76025, State: Initialized, Role: FOLLOWER
I20260928 06:28:41.291383 27560 consensus_queue.cc:260] T 00000000000000000000000000000000 P d71bb86412684ec1a52f31347cd76025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d71bb86412684ec1a52f31347cd76025" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 42001 } }
I20260928 06:28:41.291476 27560 raft_consensus.cc:388] T 00000000000000000000000000000000 P d71bb86412684ec1a52f31347cd76025 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:28:41.291531 27560 raft_consensus.cc:482] T 00000000000000000000000000000000 P d71bb86412684ec1a52f31347cd76025 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:28:41.291601 27560 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d71bb86412684ec1a52f31347cd76025 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:41.292552 27560 raft_consensus.cc:504] T 00000000000000000000000000000000 P d71bb86412684ec1a52f31347cd76025 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d71bb86412684ec1a52f31347cd76025" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 42001 } }
I20260928 06:28:41.292681 27560 leader_election.cc:302] T 00000000000000000000000000000000 P d71bb86412684ec1a52f31347cd76025 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d71bb86412684ec1a52f31347cd76025; no voters:
I20260928 06:28:41.292835 27560 leader_election.cc:288] T 00000000000000000000000000000000 P d71bb86412684ec1a52f31347cd76025 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:28:41.292891 27565 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d71bb86412684ec1a52f31347cd76025 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:28:41.293182 27560 sys_catalog.cc:564] T 00000000000000000000000000000000 P d71bb86412684ec1a52f31347cd76025 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:28:41.293203 27565 raft_consensus.cc:686] T 00000000000000000000000000000000 P d71bb86412684ec1a52f31347cd76025 [term 1 LEADER]: Becoming Leader. State: Replica: d71bb86412684ec1a52f31347cd76025, State: Running, Role: LEADER
I20260928 06:28:41.293407 27565 consensus_queue.cc:237] T 00000000000000000000000000000000 P d71bb86412684ec1a52f31347cd76025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d71bb86412684ec1a52f31347cd76025" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 42001 } }
I20260928 06:28:41.293787 27565 sys_catalog.cc:455] T 00000000000000000000000000000000 P d71bb86412684ec1a52f31347cd76025 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d71bb86412684ec1a52f31347cd76025" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d71bb86412684ec1a52f31347cd76025" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 42001 } } }
I20260928 06:28:41.293915 27565 sys_catalog.cc:458] T 00000000000000000000000000000000 P d71bb86412684ec1a52f31347cd76025 [sys.catalog]: This master's current role is: LEADER
I20260928 06:28:41.294078 27565 sys_catalog.cc:455] T 00000000000000000000000000000000 P d71bb86412684ec1a52f31347cd76025 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d71bb86412684ec1a52f31347cd76025. Latest consensus state: current_term: 1 leader_uuid: "d71bb86412684ec1a52f31347cd76025" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d71bb86412684ec1a52f31347cd76025" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 42001 } } }
I20260928 06:28:41.294170 27565 sys_catalog.cc:458] T 00000000000000000000000000000000 P d71bb86412684ec1a52f31347cd76025 [sys.catalog]: This master's current role is: LEADER
I20260928 06:28:41.294304 27574 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:28:41.294819 27574 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:28:41.296779 27581 mvcc.cc:205] Tried to move back new op lower bound from 7334203069627809792 to 7334203069618360320. Current Snapshot: MvccSnapshot[applied={T|T < 7334203069627809792}]
I20260928 06:28:41.297454 27574 catalog_manager.cc:1347] Generated new cluster ID: 686c66a03bd84fd5a5dfeee80bd33daf
Sep 28 06:28:41 dist-test-slave-d59m krb5kdc[27480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:28:41.297610 27574 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:28:41.301918 27564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:41.289674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51278 (local address 127.22.225.190:42001)
0928 06:28:41.289859 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:41.289868 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:41.289901 (+ 33us) server_negotiation.cc:407] Connection header received
0928 06:28:41.290985 (+ 1084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:41.290995 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:41.291070 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:41.291213 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:41.292067 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:41.293488 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:41.294690 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:41.295150 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:41.295267 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:41.298146 (+ 2879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:41.298149 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:41.298150 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:41.298161 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:41.300047 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:41.300399 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:41.300402 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:41.300402 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:41.300530 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:41.300733 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:41.300734 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:41.300734 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:41.301006 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:41.301143 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:41.301595 (+ 452us) server_negotiation.cc:299] Negotiation successful
0928 06:28:41.301723 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":73,"threads_started":1}
I20260928 06:28:41.316699 27574 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:28:41.317276 27574 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:28:41.320423 27574 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d71bb86412684ec1a52f31347cd76025: Generated new TSK 0
I20260928 06:28:41.320670 27574 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.225.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab.
Entry for principal kudu/127.22.225.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab.
I20260928 06:28:41.351090 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.129:0
--local_ip_for_outbound_sockets=127.22.225.129
--webserver_interface=127.22.225.129
--webserver_port=0
--tserver_master_addrs=127.22.225.190:42001
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab
--principal=kudu/127.22.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:38693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:28:41.441895 27586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:28:41.442191 27586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:28:41.442226 27586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:28:41.443720 27586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:28:41.443776 27586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:28:41.443814 27586 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.129
I20260928 06:28:41.445811 27586 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:38693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab
--principal=kudu/127.22.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.225.129
--webserver_port=0
--tserver_master_addrs=127.22.225.190:42001
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27586
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.129
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:28:41.446110 27586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:28:41.446420 27586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:28:41.452525 27591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:41.452641 27592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:41.452950 27586 server_base.cc:1034] running on GCE node
W20260928 06:28:41.452510 27594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:41.453245 27586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:28:41.453544 27586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:28:41.454721 27586 hybrid_clock.cc:648] HybridClock initialized: now 1790576921454693 us; error 46 us; skew 500 ppm
Sep 28 06:28:41 dist-test-slave-d59m krb5kdc[27480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576921, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:28:41.457813 27586 init.cc:377] Logged in from keytab as kudu/127.22.225.129@KRBTEST.COM (short username kudu)
I20260928 06:28:41.458138 27586 fs_manager.cc:359] Metadata directory not provided
I20260928 06:28:41.458211 27586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:28:41.458366 27586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:28:41.459517 27586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/minicluster-data/ts-0/data/instance:
uuid: "004a05d797a546b1aff1780bca033841"
format_stamp: "Formatted at 2026-09-28 06:28:41 on dist-test-slave-d59m"
server_key: "0b75ab78e516e8b96e386a26dad91e2b"
server_key_iv: "aaab39d9da8b99dd3953b04857949415"
server_key_version: "encryptionkey@0"
I20260928 06:28:41.459872 27586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/minicluster-data/ts-0/wal/instance:
uuid: "004a05d797a546b1aff1780bca033841"
format_stamp: "Formatted at 2026-09-28 06:28:41 on dist-test-slave-d59m"
server_key: "0b75ab78e516e8b96e386a26dad91e2b"
server_key_iv: "aaab39d9da8b99dd3953b04857949415"
server_key_version: "encryptionkey@0"
I20260928 06:28:41.461272 27586 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:28:41.462042 27600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:41.462209 27586 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:28:41.462304 27586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/minicluster-data/ts-0/wal
uuid: "004a05d797a546b1aff1780bca033841"
format_stamp: "Formatted at 2026-09-28 06:28:41 on dist-test-slave-d59m"
server_key: "0b75ab78e516e8b96e386a26dad91e2b"
server_key_iv: "aaab39d9da8b99dd3953b04857949415"
server_key_version: "encryptionkey@0"
I20260928 06:28:41.462383 27586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:28:41.477734 27586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:28:41.480103 27586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:28:41.480260 27586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:28:41.480499 27586 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:28:41.480826 27586 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:28:41.480878 27586 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:41.480918 27586 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:28:41.480945 27586 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:41.486846 27586 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.129:35071
I20260928 06:28:41.486882 27713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.129:35071 every 8 connection(s)
I20260928 06:28:41.487159 27586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
Sep 28 06:28:41 dist-test-slave-d59m krb5kdc[27480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576921, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.129@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:28:41.496618 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 27586
I20260928 06:28:41.496738 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/minicluster-data/ts-0/wal/instance
I20260928 06:28:41.496924 23430 external_mini_cluster.cc:1428] Setting key 215f8152cf3cc2934412400cf0f33401
I20260928 06:28:41.496979 27564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:41.487924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:48257 (local address 127.22.225.190:42001)
0928 06:28:41.487988 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:41.487990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:41.488200 (+ 210us) server_negotiation.cc:407] Connection header received
0928 06:28:41.488764 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:41.488765 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:41.488785 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:41.488853 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:41.489769 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:41.490564 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:41.491514 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:41.491794 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:41.491864 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:41.494142 (+ 2278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:41.494144 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:41.494144 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:41.494152 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:41.495593 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:41.495968 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:41.495969 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:41.495970 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:41.496037 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:41.496251 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:41.496252 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:41.496252 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:41.496378 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:41.496442 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:41.496811 (+ 369us) server_negotiation.cc:299] Negotiation successful
0928 06:28:41.496899 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 06:28:41.497363 27716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:41.487898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:42001 (local address 127.22.225.129:48257)
0928 06:28:41.488085 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:41.488121 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:41.488660 (+ 539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:41.488904 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:41.488908 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:41.489163 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:41.489672 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:41.489676 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:41.490653 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:41.490654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:41.491431 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:41.491433 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:41.491874 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:41.491875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:41.491995 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:41.492525 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:41.492533 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:41.494054 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:41.495679 (+ 1625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:41.495685 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:41.495686 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:41.495898 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:41.496100 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:41.496101 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:41.496101 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:41.496192 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:41.496461 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:41.496467 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:41.496661 (+ 194us) client_negotiation.cc:769] Sending connection context
0928 06:28:41.496787 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 06:28:41.496956 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":122,"threads_started":1}
I20260928 06:28:41.497814 27714 heartbeater.cc:347] Connected to a master server at 127.22.225.190:42001
I20260928 06:28:41.497912 27714 heartbeater.cc:464] Registering TS with master...
I20260928 06:28:41.498092 27714 heartbeater.cc:511] Master 127.22.225.190:42001 requested a full tablet report, sending...
I20260928 06:28:41.498621 27524 ts_manager.cc:194] Registered new tserver with Master: 004a05d797a546b1aff1780bca033841 (127.22.225.129:35071)
I20260928 06:28:41.499543 27524 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.225.129@KRBTEST.COM'} at 127.22.225.129:48257
WARNING: no policy specified for kudu/127.22.225.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab.
Entry for principal kudu/127.22.225.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab.
I20260928 06:28:41.523128 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.130:0
--local_ip_for_outbound_sockets=127.22.225.130
--webserver_interface=127.22.225.130
--webserver_port=0
--tserver_master_addrs=127.22.225.190:42001
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab
--principal=kudu/127.22.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:38693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:28:41.604557 27719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:28:41.604846 27719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:28:41.604887 27719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:28:41.606074 27719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:28:41.606141 27719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:28:41.606189 27719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.130
I20260928 06:28:41.607707 27719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:38693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab
--principal=kudu/127.22.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.225.130
--webserver_port=0
--tserver_master_addrs=127.22.225.190:42001
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.130
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:28:41.607964 27719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:28:41.608217 27719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:28:41.613054 27724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:41.613132 27727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:41.613235 27719 server_base.cc:1034] running on GCE node
W20260928 06:28:41.613054 27725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:41.613668 27719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:28:41.613929 27719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:28:41.615073 27719 hybrid_clock.cc:648] HybridClock initialized: now 1790576921615064 us; error 35 us; skew 500 ppm
Sep 28 06:28:41 dist-test-slave-d59m krb5kdc[27480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576921, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:28:41.617614 27719 init.cc:377] Logged in from keytab as kudu/127.22.225.130@KRBTEST.COM (short username kudu)
I20260928 06:28:41.618003 27719 fs_manager.cc:359] Metadata directory not provided
I20260928 06:28:41.618052 27719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:28:41.618213 27719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:28:41.619261 27719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/minicluster-data/ts-1/data/instance:
uuid: "4ed4756c1d924686834e7b22938f9329"
format_stamp: "Formatted at 2026-09-28 06:28:41 on dist-test-slave-d59m"
server_key: "33db35a15a45e6066e65f3b2f8913052"
server_key_iv: "151ac3a0388f1a0eed15bd515648b049"
server_key_version: "encryptionkey@0"
I20260928 06:28:41.619612 27719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/minicluster-data/ts-1/wal/instance:
uuid: "4ed4756c1d924686834e7b22938f9329"
format_stamp: "Formatted at 2026-09-28 06:28:41 on dist-test-slave-d59m"
server_key: "33db35a15a45e6066e65f3b2f8913052"
server_key_iv: "151ac3a0388f1a0eed15bd515648b049"
server_key_version: "encryptionkey@0"
I20260928 06:28:41.620918 27719 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:28:41.621654 27733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:41.621802 27719 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:41.621876 27719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/minicluster-data/ts-1/wal
uuid: "4ed4756c1d924686834e7b22938f9329"
format_stamp: "Formatted at 2026-09-28 06:28:41 on dist-test-slave-d59m"
server_key: "33db35a15a45e6066e65f3b2f8913052"
server_key_iv: "151ac3a0388f1a0eed15bd515648b049"
server_key_version: "encryptionkey@0"
I20260928 06:28:41.621943 27719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:28:41.638578 27719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:28:41.641032 27719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:28:41.641177 27719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:28:41.641454 27719 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:28:41.641839 27719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:28:41.641875 27719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:41.641897 27719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:28:41.641911 27719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:41.649237 27719 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.130:44137
I20260928 06:28:41.649310 27846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.130:44137 every 8 connection(s)
I20260928 06:28:41.649600 27719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
Sep 28 06:28:41 dist-test-slave-d59m krb5kdc[27480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576921, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.130@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:28:41.658190 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 27719
I20260928 06:28:41.658315 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/minicluster-data/ts-1/wal/instance
I20260928 06:28:41.658519 23430 external_mini_cluster.cc:1428] Setting key 19f11f8b706fcc2c444fd998d2bb1a78
I20260928 06:28:41.660305 27564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:41.650381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:60901 (local address 127.22.225.190:42001)
0928 06:28:41.650457 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:41.650459 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:41.650714 (+ 255us) server_negotiation.cc:407] Connection header received
0928 06:28:41.651338 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:41.651340 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:41.651353 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:41.651423 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:41.652700 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:41.653217 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:41.654572 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:41.654744 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:41.654796 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:41.657157 (+ 2361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:41.657158 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:41.657159 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:41.657166 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:41.658649 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:41.659104 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:41.659106 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:41.659106 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:41.659174 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:41.659494 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:41.659495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:41.659496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:41.659665 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:41.659738 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:41.660069 (+ 331us) server_negotiation.cc:299] Negotiation successful
0928 06:28:41.660164 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 06:28:41.660410 27849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:41.650346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:42001 (local address 127.22.225.130:60901)
0928 06:28:41.650551 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:41.650614 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:41.651218 (+ 604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:41.651507 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:41.651512 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:41.651873 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:41.652605 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:41.652613 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:41.653313 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:41.653314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:41.654487 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:41.654490 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:41.654840 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:41.654841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:41.654979 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:41.655517 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:41.655528 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:41.657062 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:41.658732 (+ 1670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:41.658739 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:41.658741 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:41.659022 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:41.659267 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:41.659268 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:41.659269 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:41.659414 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:41.659742 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:41.659748 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:41.659917 (+ 169us) client_negotiation.cc:769] Sending connection context
0928 06:28:41.660034 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 06:28:41.660191 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":106,"threads_started":1}
I20260928 06:28:41.661149 27847 heartbeater.cc:347] Connected to a master server at 127.22.225.190:42001
I20260928 06:28:41.661257 27847 heartbeater.cc:464] Registering TS with master...
I20260928 06:28:41.661419 27847 heartbeater.cc:511] Master 127.22.225.190:42001 requested a full tablet report, sending...
I20260928 06:28:41.661854 27524 ts_manager.cc:194] Registered new tserver with Master: 4ed4756c1d924686834e7b22938f9329 (127.22.225.130:44137)
I20260928 06:28:41.662297 27524 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.225.130@KRBTEST.COM'} at 127.22.225.130:60901
WARNING: no policy specified for kudu/127.22.225.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab.
Entry for principal kudu/127.22.225.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab.
I20260928 06:28:41.683696 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.131:0
--local_ip_for_outbound_sockets=127.22.225.131
--webserver_interface=127.22.225.131
--webserver_port=0
--tserver_master_addrs=127.22.225.190:42001
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab
--principal=kudu/127.22.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:38693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:28:41.766301 27852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:28:41.766516 27852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:28:41.766556 27852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:28:41.767753 27852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:28:41.767822 27852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:28:41.767865 27852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.131
I20260928 06:28:41.769492 27852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:38693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab
--principal=kudu/127.22.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.225.131
--webserver_port=0
--tserver_master_addrs=127.22.225.190:42001
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.131
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:28:41.769752 27852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:28:41.769972 27852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:28:41.774690 27858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:41.774870 27852 server_base.cc:1034] running on GCE node
W20260928 06:28:41.774694 27860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:41.774742 27857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:41.775234 27852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:28:41.775503 27852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:28:41.776635 27852 hybrid_clock.cc:648] HybridClock initialized: now 1790576921776620 us; error 29 us; skew 500 ppm
Sep 28 06:28:41 dist-test-slave-d59m krb5kdc[27480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576921, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:28:41.779237 27852 init.cc:377] Logged in from keytab as kudu/127.22.225.131@KRBTEST.COM (short username kudu)
I20260928 06:28:41.779615 27852 fs_manager.cc:359] Metadata directory not provided
I20260928 06:28:41.779677 27852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:28:41.779799 27852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:28:41.781121 27852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/minicluster-data/ts-2/data/instance:
uuid: "9c1c6f41f79c4ad3bf26379397d12f9a"
format_stamp: "Formatted at 2026-09-28 06:28:41 on dist-test-slave-d59m"
server_key: "6a463c4fda5e898b5e1cdbaef760ee59"
server_key_iv: "20f66c4f39e6f9b55a5628e691236cf2"
server_key_version: "encryptionkey@0"
I20260928 06:28:41.781520 27852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/minicluster-data/ts-2/wal/instance:
uuid: "9c1c6f41f79c4ad3bf26379397d12f9a"
format_stamp: "Formatted at 2026-09-28 06:28:41 on dist-test-slave-d59m"
server_key: "6a463c4fda5e898b5e1cdbaef760ee59"
server_key_iv: "20f66c4f39e6f9b55a5628e691236cf2"
server_key_version: "encryptionkey@0"
I20260928 06:28:41.783041 27852 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:28:41.783834 27866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:41.784099 27852 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:28:41.784196 27852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/minicluster-data/ts-2/wal
uuid: "9c1c6f41f79c4ad3bf26379397d12f9a"
format_stamp: "Formatted at 2026-09-28 06:28:41 on dist-test-slave-d59m"
server_key: "6a463c4fda5e898b5e1cdbaef760ee59"
server_key_iv: "20f66c4f39e6f9b55a5628e691236cf2"
server_key_version: "encryptionkey@0"
I20260928 06:28:41.784264 27852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:28:41.801126 27852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:28:41.803511 27852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:28:41.803661 27852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:28:41.803912 27852 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:28:41.804239 27852 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:28:41.804282 27852 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:41.804311 27852 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:28:41.804337 27852 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:41.812021 27852 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.131:39345
I20260928 06:28:41.812062 27979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.131:39345 every 8 connection(s)
I20260928 06:28:41.812427 27852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
I20260928 06:28:41.818970 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 27852
I20260928 06:28:41.819080 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/minicluster-data/ts-2/wal/instance
Sep 28 06:28:41 dist-test-slave-d59m krb5kdc[27480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576921, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.131@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:28:41.819283 23430 external_mini_cluster.cc:1428] Setting key 406c1665f074a3a17436f184dd4ac473
I20260928 06:28:41.822695 27564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:41.813204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:49669 (local address 127.22.225.190:42001)
0928 06:28:41.813264 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:41.813266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:41.813502 (+ 236us) server_negotiation.cc:407] Connection header received
0928 06:28:41.814022 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:41.814025 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:41.814040 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:41.814151 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:41.815037 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:41.815830 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:41.816840 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:41.817192 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:41.817264 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:41.819704 (+ 2440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:41.819705 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:41.819706 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:41.819714 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:41.821370 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:41.821770 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:41.821771 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:41.821772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:41.821825 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:41.822079 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:41.822079 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:41.822080 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:41.822194 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:41.822260 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:41.822530 (+ 270us) server_negotiation.cc:299] Negotiation successful
0928 06:28:41.822606 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 06:28:41.822924 27982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:41.813169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:42001 (local address 127.22.225.131:49669)
0928 06:28:41.813334 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:41.813379 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:41.813874 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:41.814190 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:41.814193 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:41.814444 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:41.814950 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:41.814954 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:41.815929 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:41.815930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:41.816761 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:41.816763 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:41.817280 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:41.817281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:41.817373 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:41.817936 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:41.817944 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:41.819605 (+ 1661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:41.821458 (+ 1853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:41.821465 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:41.821467 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:41.821681 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:41.821892 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:41.821893 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:41.821893 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:41.822005 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:41.822262 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:41.822268 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:41.822401 (+ 133us) client_negotiation.cc:769] Sending connection context
0928 06:28:41.822508 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 06:28:41.822654 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":95,"threads_started":1}
I20260928 06:28:41.823381 27980 heartbeater.cc:347] Connected to a master server at 127.22.225.190:42001
I20260928 06:28:41.823485 27980 heartbeater.cc:464] Registering TS with master...
I20260928 06:28:41.823642 27980 heartbeater.cc:511] Master 127.22.225.190:42001 requested a full tablet report, sending...
I20260928 06:28:41.824018 27524 ts_manager.cc:194] Registered new tserver with Master: 9c1c6f41f79c4ad3bf26379397d12f9a (127.22.225.131:39345)
I20260928 06:28:41.824565 27524 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.225.131@KRBTEST.COM'} at 127.22.225.131:49669
I20260928 06:28:41.831195 23430 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:28:41.838328 27564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:41.832453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51282 (local address 127.22.225.190:42001)
0928 06:28:41.832493 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:41.832495 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:41.832615 (+ 120us) server_negotiation.cc:407] Connection header received
0928 06:28:41.832671 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:41.832672 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:41.832681 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:41.832738 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:41.833238 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:41.833810 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:41.834489 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:41.834706 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:41.834772 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:41.835835 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:41.835836 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:41.835836 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:41.835842 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:41.837292 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:41.837594 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:41.837594 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:41.837595 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:41.837652 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:41.837815 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:41.837815 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:41.837816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:41.837994 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:41.838106 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:41.838183 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 06:28:41.838252 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:28:41.839974 27524 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51282:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:28:41.841939 27524 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:28:41.848910 27991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:41.844231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.130:44137 (local address 127.0.0.1:45766)
0928 06:28:41.844633 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:41.844675 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:41.844742 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:41.845481 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:41.845482 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:41.845484 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:28:41.845532 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:41.845534 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:41.846693 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:41.846694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:41.847733 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:41.847735 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:41.848632 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:41.848633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:41.848711 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:41.848724 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 06:28:41.848788 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 06:28:41.848832 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":62,"threads_started":1}
I20260928 06:28:41.849128 27993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:41.844335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45766 (local address 127.22.225.130:44137)
0928 06:28:41.845331 (+ 996us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:41.845334 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:41.845344 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:28:41.845377 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:41.845379 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:41.845392 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:41.845470 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:28:41.845616 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:41.846580 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:41.847821 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:41.848458 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:41.848526 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:41.848577 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:41.849020 (+ 443us) server_negotiation.cc:299] Negotiation successful
0928 06:28:41.849059 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":958,"thread_start_us":87,"threads_started":1}
I20260928 06:28:41.849684 27992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:41.844470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.131:39345 (local address 127.0.0.1:37438)
0928 06:28:41.845165 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:41.845175 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:41.845217 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:41.845755 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:41.845756 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:41.845757 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:28:41.845795 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:41.845796 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:41.847130 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:41.847131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:41.848178 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:41.848180 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:41.849245 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:41.849246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:41.849520 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:41.849525 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:28:41.849556 (+ 31us) client_negotiation.cc:240] Negotiation successful
0928 06:28:41.849595 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"mutex_wait_us":28,"thread_start_us":75,"threads_started":1}
I20260928 06:28:41.849886 27781 tablet_service.cc:1467] Processing CreateTablet for tablet e954cbe48b6048ac93fa425d0175fda4 (DEFAULT_TABLE table=test-table [id=337a0d2e756a48a68cbf9f743a452037]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:28:41.850037 27996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:41.844642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44868 (local address 127.22.225.129:35071)
0928 06:28:41.844831 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:41.844836 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:41.844845 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:28:41.844886 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:41.844888 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:41.844900 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:41.844985 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:28:41.845224 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:41.846348 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:41.847940 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:41.848704 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:41.848770 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:41.848817 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:41.849829 (+ 1012us) server_negotiation.cc:299] Negotiation successful
0928 06:28:41.849872 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":85,"threads_started":1}
I20260928 06:28:41.849782 27994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:41.844310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.129:35071 (local address 127.0.0.1:44868)
0928 06:28:41.844670 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:41.844682 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:41.844743 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:41.844996 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:41.845000 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:41.845002 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:28:41.845080 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:41.845082 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:41.846454 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:41.846455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:41.847838 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:41.847840 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:41.848849 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:41.848850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:41.848915 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:41.848919 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:28:41.849670 (+ 751us) client_negotiation.cc:240] Negotiation successful
0928 06:28:41.849709 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":96,"threads_started":1}
I20260928 06:28:41.850143 27995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:41.844606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37438 (local address 127.22.225.131:39345)
0928 06:28:41.845606 (+ 1000us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:41.845610 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:41.845618 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:28:41.845649 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:41.845651 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:41.845665 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:41.845899 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:28:41.845970 (+ 71us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:41.847013 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:41.848266 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:41.849130 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:41.849939 (+ 809us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:41.849995 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:41.850018 (+ 23us) server_negotiation.cc:299] Negotiation successful
0928 06:28:41.850058 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":970,"thread_start_us":89,"threads_started":1}
I20260928 06:28:41.850224 27781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e954cbe48b6048ac93fa425d0175fda4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:41.850706 27914 tablet_service.cc:1467] Processing CreateTablet for tablet e954cbe48b6048ac93fa425d0175fda4 (DEFAULT_TABLE table=test-table [id=337a0d2e756a48a68cbf9f743a452037]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:28:41.850706 27648 tablet_service.cc:1467] Processing CreateTablet for tablet e954cbe48b6048ac93fa425d0175fda4 (DEFAULT_TABLE table=test-table [id=337a0d2e756a48a68cbf9f743a452037]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:28:41.850996 27914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e954cbe48b6048ac93fa425d0175fda4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:41.851007 27648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e954cbe48b6048ac93fa425d0175fda4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:41.854599 27998 tablet_bootstrap.cc:492] T e954cbe48b6048ac93fa425d0175fda4 P 9c1c6f41f79c4ad3bf26379397d12f9a: Bootstrap starting.
I20260928 06:28:41.855249 27998 tablet_bootstrap.cc:654] T e954cbe48b6048ac93fa425d0175fda4 P 9c1c6f41f79c4ad3bf26379397d12f9a: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:41.855535 27999 tablet_bootstrap.cc:492] T e954cbe48b6048ac93fa425d0175fda4 P 004a05d797a546b1aff1780bca033841: Bootstrap starting.
I20260928 06:28:41.855870 27998 log.cc:824] T e954cbe48b6048ac93fa425d0175fda4 P 9c1c6f41f79c4ad3bf26379397d12f9a: Log is configured to *not* fsync() on all Append() calls
I20260928 06:28:41.855827 27997 tablet_bootstrap.cc:492] T e954cbe48b6048ac93fa425d0175fda4 P 4ed4756c1d924686834e7b22938f9329: Bootstrap starting.
I20260928 06:28:41.856264 27999 tablet_bootstrap.cc:654] T e954cbe48b6048ac93fa425d0175fda4 P 004a05d797a546b1aff1780bca033841: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:41.856612 27998 tablet_bootstrap.cc:492] T e954cbe48b6048ac93fa425d0175fda4 P 9c1c6f41f79c4ad3bf26379397d12f9a: No bootstrap required, opened a new log
I20260928 06:28:41.856680 27998 ts_tablet_manager.cc:1397] T e954cbe48b6048ac93fa425d0175fda4 P 9c1c6f41f79c4ad3bf26379397d12f9a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:28:41.856729 27997 tablet_bootstrap.cc:654] T e954cbe48b6048ac93fa425d0175fda4 P 4ed4756c1d924686834e7b22938f9329: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:41.856980 27999 log.cc:824] T e954cbe48b6048ac93fa425d0175fda4 P 004a05d797a546b1aff1780bca033841: Log is configured to *not* fsync() on all Append() calls
I20260928 06:28:41.857281 27997 log.cc:824] T e954cbe48b6048ac93fa425d0175fda4 P 4ed4756c1d924686834e7b22938f9329: Log is configured to *not* fsync() on all Append() calls
I20260928 06:28:41.858050 27997 tablet_bootstrap.cc:492] T e954cbe48b6048ac93fa425d0175fda4 P 4ed4756c1d924686834e7b22938f9329: No bootstrap required, opened a new log
I20260928 06:28:41.858121 27997 ts_tablet_manager.cc:1397] T e954cbe48b6048ac93fa425d0175fda4 P 4ed4756c1d924686834e7b22938f9329: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:28:41.858839 27998 raft_consensus.cc:348] T e954cbe48b6048ac93fa425d0175fda4 P 9c1c6f41f79c4ad3bf26379397d12f9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed4756c1d924686834e7b22938f9329" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 44137 } } peers { permanent_uuid: "9c1c6f41f79c4ad3bf26379397d12f9a" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 39345 } } peers { permanent_uuid: "004a05d797a546b1aff1780bca033841" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 35071 } }
I20260928 06:28:41.859007 27998 raft_consensus.cc:374] T e954cbe48b6048ac93fa425d0175fda4 P 9c1c6f41f79c4ad3bf26379397d12f9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:41.859043 27998 raft_consensus.cc:729] T e954cbe48b6048ac93fa425d0175fda4 P 9c1c6f41f79c4ad3bf26379397d12f9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c1c6f41f79c4ad3bf26379397d12f9a, State: Initialized, Role: FOLLOWER
I20260928 06:28:41.859155 27998 consensus_queue.cc:260] T e954cbe48b6048ac93fa425d0175fda4 P 9c1c6f41f79c4ad3bf26379397d12f9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed4756c1d924686834e7b22938f9329" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 44137 } } peers { permanent_uuid: "9c1c6f41f79c4ad3bf26379397d12f9a" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 39345 } } peers { permanent_uuid: "004a05d797a546b1aff1780bca033841" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 35071 } }
I20260928 06:28:41.859436 27980 heartbeater.cc:503] Master 127.22.225.190:42001 was elected leader, sending a full tablet report...
I20260928 06:28:41.859674 27998 ts_tablet_manager.cc:1428] T e954cbe48b6048ac93fa425d0175fda4 P 9c1c6f41f79c4ad3bf26379397d12f9a: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 06:28:41.860121 27997 raft_consensus.cc:348] T e954cbe48b6048ac93fa425d0175fda4 P 4ed4756c1d924686834e7b22938f9329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed4756c1d924686834e7b22938f9329" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 44137 } } peers { permanent_uuid: "9c1c6f41f79c4ad3bf26379397d12f9a" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 39345 } } peers { permanent_uuid: "004a05d797a546b1aff1780bca033841" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 35071 } }
I20260928 06:28:41.860270 27997 raft_consensus.cc:374] T e954cbe48b6048ac93fa425d0175fda4 P 4ed4756c1d924686834e7b22938f9329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:41.860303 27997 raft_consensus.cc:729] T e954cbe48b6048ac93fa425d0175fda4 P 4ed4756c1d924686834e7b22938f9329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ed4756c1d924686834e7b22938f9329, State: Initialized, Role: FOLLOWER
I20260928 06:28:41.860410 27997 consensus_queue.cc:260] T e954cbe48b6048ac93fa425d0175fda4 P 4ed4756c1d924686834e7b22938f9329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed4756c1d924686834e7b22938f9329" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 44137 } } peers { permanent_uuid: "9c1c6f41f79c4ad3bf26379397d12f9a" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 39345 } } peers { permanent_uuid: "004a05d797a546b1aff1780bca033841" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 35071 } }
I20260928 06:28:41.861001 27997 ts_tablet_manager.cc:1428] T e954cbe48b6048ac93fa425d0175fda4 P 4ed4756c1d924686834e7b22938f9329: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260928 06:28:41.861081 27999 tablet_bootstrap.cc:492] T e954cbe48b6048ac93fa425d0175fda4 P 004a05d797a546b1aff1780bca033841: No bootstrap required, opened a new log
I20260928 06:28:41.861116 27847 heartbeater.cc:503] Master 127.22.225.190:42001 was elected leader, sending a full tablet report...
I20260928 06:28:41.861145 27999 ts_tablet_manager.cc:1397] T e954cbe48b6048ac93fa425d0175fda4 P 004a05d797a546b1aff1780bca033841: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260928 06:28:41.862784 27999 raft_consensus.cc:348] T e954cbe48b6048ac93fa425d0175fda4 P 004a05d797a546b1aff1780bca033841 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed4756c1d924686834e7b22938f9329" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 44137 } } peers { permanent_uuid: "9c1c6f41f79c4ad3bf26379397d12f9a" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 39345 } } peers { permanent_uuid: "004a05d797a546b1aff1780bca033841" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 35071 } }
I20260928 06:28:41.862902 27999 raft_consensus.cc:374] T e954cbe48b6048ac93fa425d0175fda4 P 004a05d797a546b1aff1780bca033841 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:41.862923 27999 raft_consensus.cc:729] T e954cbe48b6048ac93fa425d0175fda4 P 004a05d797a546b1aff1780bca033841 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 004a05d797a546b1aff1780bca033841, State: Initialized, Role: FOLLOWER
I20260928 06:28:41.863044 27999 consensus_queue.cc:260] T e954cbe48b6048ac93fa425d0175fda4 P 004a05d797a546b1aff1780bca033841 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed4756c1d924686834e7b22938f9329" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 44137 } } peers { permanent_uuid: "9c1c6f41f79c4ad3bf26379397d12f9a" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 39345 } } peers { permanent_uuid: "004a05d797a546b1aff1780bca033841" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 35071 } }
I20260928 06:28:41.863330 27999 ts_tablet_manager.cc:1428] T e954cbe48b6048ac93fa425d0175fda4 P 004a05d797a546b1aff1780bca033841: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:28:41.863327 27714 heartbeater.cc:503] Master 127.22.225.190:42001 was elected leader, sending a full tablet report...
W20260928 06:28:41.900164 27848 tablet.cc:2369] T e954cbe48b6048ac93fa425d0175fda4 P 4ed4756c1d924686834e7b22938f9329: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:28:41.921629 28005 raft_consensus.cc:482] T e954cbe48b6048ac93fa425d0175fda4 P 004a05d797a546b1aff1780bca033841 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:28:41.921792 28005 raft_consensus.cc:504] T e954cbe48b6048ac93fa425d0175fda4 P 004a05d797a546b1aff1780bca033841 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed4756c1d924686834e7b22938f9329" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 44137 } } peers { permanent_uuid: "9c1c6f41f79c4ad3bf26379397d12f9a" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 39345 } } peers { permanent_uuid: "004a05d797a546b1aff1780bca033841" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 35071 } }
I20260928 06:28:41.922176 28005 leader_election.cc:288] T e954cbe48b6048ac93fa425d0175fda4 P 004a05d797a546b1aff1780bca033841 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ed4756c1d924686834e7b22938f9329 (127.22.225.130:44137), 9c1c6f41f79c4ad3bf26379397d12f9a (127.22.225.131:39345)
I20260928 06:28:41.925848 27716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:41.922372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.130:44137 (local address 127.22.225.129:37213)
0928 06:28:41.922438 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:41.922452 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:41.922517 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:41.922738 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:41.922739 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:41.922741 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:28:41.922817 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:41.922820 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:41.923744 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:41.923746 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:41.924843 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:41.924855 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:41.925571 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:41.925572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:41.925659 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:41.925663 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:28:41.925718 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 06:28:41.925763 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":18}
I20260928 06:28:41.925861 27993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:41.922464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:37213 (local address 127.22.225.130:44137)
0928 06:28:41.922515 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:41.922517 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:41.922527 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:28:41.922610 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:41.922611 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:41.922621 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:41.922676 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:28:41.922891 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:41.923653 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:41.924923 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:41.925453 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:41.925525 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:41.925535 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:41.925736 (+ 201us) server_negotiation.cc:299] Negotiation successful
0928 06:28:41.925781 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:28:41.925912 27995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:41.922657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:57777 (local address 127.22.225.131:39345)
0928 06:28:41.922698 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:41.922700 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:41.922708 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:28:41.922765 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:41.922765 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:41.922774 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:41.922835 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:28:41.923013 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:41.923731 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:41.924893 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:41.925471 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:41.925550 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:41.925557 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:41.925777 (+ 220us) server_negotiation.cc:299] Negotiation successful
0928 06:28:41.925812 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 06:28:41.926270 27801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e954cbe48b6048ac93fa425d0175fda4" candidate_uuid: "004a05d797a546b1aff1780bca033841" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ed4756c1d924686834e7b22938f9329" is_pre_election: true
I20260928 06:28:41.926419 27801 raft_consensus.cc:2454] T e954cbe48b6048ac93fa425d0175fda4 P 4ed4756c1d924686834e7b22938f9329 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 004a05d797a546b1aff1780bca033841 in term 0.
I20260928 06:28:41.926600 28006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:41.922475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.131:39345 (local address 127.22.225.129:57777)
0928 06:28:41.922647 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:41.922658 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:41.922712 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:41.922862 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:41.922863 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:41.922865 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:28:41.922936 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:41.922937 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:41.923793 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:41.923794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:41.924796 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:41.924799 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:41.925627 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:41.925628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:41.925699 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:41.925702 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:28:41.926449 (+ 747us) client_negotiation.cc:240] Negotiation successful
0928 06:28:41.926516 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":120,"threads_started":1}
I20260928 06:28:41.926633 27602 leader_election.cc:302] T e954cbe48b6048ac93fa425d0175fda4 P 004a05d797a546b1aff1780bca033841 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 004a05d797a546b1aff1780bca033841, 4ed4756c1d924686834e7b22938f9329; no voters:
I20260928 06:28:41.926743 28005 raft_consensus.cc:2781] T e954cbe48b6048ac93fa425d0175fda4 P 004a05d797a546b1aff1780bca033841 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:28:41.926791 28005 raft_consensus.cc:482] T e954cbe48b6048ac93fa425d0175fda4 P 004a05d797a546b1aff1780bca033841 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:28:41.926819 28005 raft_consensus.cc:3037] T e954cbe48b6048ac93fa425d0175fda4 P 004a05d797a546b1aff1780bca033841 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:41.926991 27934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e954cbe48b6048ac93fa425d0175fda4" candidate_uuid: "004a05d797a546b1aff1780bca033841" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c1c6f41f79c4ad3bf26379397d12f9a" is_pre_election: true
I20260928 06:28:41.927129 27934 raft_consensus.cc:2454] T e954cbe48b6048ac93fa425d0175fda4 P 9c1c6f41f79c4ad3bf26379397d12f9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 004a05d797a546b1aff1780bca033841 in term 0.
I20260928 06:28:41.927657 28005 raft_consensus.cc:504] T e954cbe48b6048ac93fa425d0175fda4 P 004a05d797a546b1aff1780bca033841 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed4756c1d924686834e7b22938f9329" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 44137 } } peers { permanent_uuid: "9c1c6f41f79c4ad3bf26379397d12f9a" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 39345 } } peers { permanent_uuid: "004a05d797a546b1aff1780bca033841" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 35071 } }
I20260928 06:28:41.927796 28005 leader_election.cc:288] T e954cbe48b6048ac93fa425d0175fda4 P 004a05d797a546b1aff1780bca033841 [CANDIDATE]: Term 1 election: Requested vote from peers 4ed4756c1d924686834e7b22938f9329 (127.22.225.130:44137), 9c1c6f41f79c4ad3bf26379397d12f9a (127.22.225.131:39345)
I20260928 06:28:41.928001 27934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e954cbe48b6048ac93fa425d0175fda4" candidate_uuid: "004a05d797a546b1aff1780bca033841" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c1c6f41f79c4ad3bf26379397d12f9a"
I20260928 06:28:41.928078 27934 raft_consensus.cc:3037] T e954cbe48b6048ac93fa425d0175fda4 P 9c1c6f41f79c4ad3bf26379397d12f9a [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:41.928944 27934 raft_consensus.cc:2454] T e954cbe48b6048ac93fa425d0175fda4 P 9c1c6f41f79c4ad3bf26379397d12f9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 004a05d797a546b1aff1780bca033841 in term 1.
I20260928 06:28:41.929049 27801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e954cbe48b6048ac93fa425d0175fda4" candidate_uuid: "004a05d797a546b1aff1780bca033841" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ed4756c1d924686834e7b22938f9329"
I20260928 06:28:41.929121 27801 raft_consensus.cc:3037] T e954cbe48b6048ac93fa425d0175fda4 P 4ed4756c1d924686834e7b22938f9329 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:41.929136 27602 leader_election.cc:302] T e954cbe48b6048ac93fa425d0175fda4 P 004a05d797a546b1aff1780bca033841 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 004a05d797a546b1aff1780bca033841, 9c1c6f41f79c4ad3bf26379397d12f9a; no voters:
I20260928 06:28:41.929225 28005 raft_consensus.cc:2781] T e954cbe48b6048ac93fa425d0175fda4 P 004a05d797a546b1aff1780bca033841 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:28:41.929404 28005 raft_consensus.cc:686] T e954cbe48b6048ac93fa425d0175fda4 P 004a05d797a546b1aff1780bca033841 [term 1 LEADER]: Becoming Leader. State: Replica: 004a05d797a546b1aff1780bca033841, State: Running, Role: LEADER
I20260928 06:28:41.929510 28005 consensus_queue.cc:237] T e954cbe48b6048ac93fa425d0175fda4 P 004a05d797a546b1aff1780bca033841 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed4756c1d924686834e7b22938f9329" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 44137 } } peers { permanent_uuid: "9c1c6f41f79c4ad3bf26379397d12f9a" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 39345 } } peers { permanent_uuid: "004a05d797a546b1aff1780bca033841" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 35071 } }
I20260928 06:28:41.929952 27801 raft_consensus.cc:2454] T e954cbe48b6048ac93fa425d0175fda4 P 4ed4756c1d924686834e7b22938f9329 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 004a05d797a546b1aff1780bca033841 in term 1.
I20260928 06:28:41.930503 27523 catalog_manager.cc:5495] T e954cbe48b6048ac93fa425d0175fda4 P 004a05d797a546b1aff1780bca033841 reported cstate change: term changed from 0 to 1, leader changed from <none> to 004a05d797a546b1aff1780bca033841 (127.22.225.129). New cstate: current_term: 1 leader_uuid: "004a05d797a546b1aff1780bca033841" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed4756c1d924686834e7b22938f9329" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 44137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c1c6f41f79c4ad3bf26379397d12f9a" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 39345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "004a05d797a546b1aff1780bca033841" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 35071 } health_report { overall_health: HEALTHY } } }
I20260928 06:28:41.940672 27996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:41.938253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44874 (local address 127.22.225.129:35071)
0928 06:28:41.938296 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:41.938299 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:41.938308 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:28:41.938383 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:41.938384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:41.938401 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:41.938484 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:28:41.938628 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:41.939091 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:41.939854 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:41.940059 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:41.940131 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:41.940304 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:28:41.940409 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:28:41.940498 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:41.940537 (+ 39us) server_negotiation.cc:299] Negotiation successful
0928 06:28:41.940555 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 06:28:41.942425 27801 raft_consensus.cc:1261] T e954cbe48b6048ac93fa425d0175fda4 P 4ed4756c1d924686834e7b22938f9329 [term 1 FOLLOWER]: Refusing update from remote peer 004a05d797a546b1aff1780bca033841: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:28:41.942435 27934 raft_consensus.cc:1261] T e954cbe48b6048ac93fa425d0175fda4 P 9c1c6f41f79c4ad3bf26379397d12f9a [term 1 FOLLOWER]: Refusing update from remote peer 004a05d797a546b1aff1780bca033841: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:28:41.942756 28005 consensus_queue.cc:1035] T e954cbe48b6048ac93fa425d0175fda4 P 004a05d797a546b1aff1780bca033841 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c1c6f41f79c4ad3bf26379397d12f9a" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 39345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:28:41.942890 28007 consensus_queue.cc:1035] T e954cbe48b6048ac93fa425d0175fda4 P 004a05d797a546b1aff1780bca033841 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ed4756c1d924686834e7b22938f9329" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:28:41.944967 28010 mvcc.cc:205] Tried to move back new op lower bound from 7334203072273559552 to 7334203072224030720. Current Snapshot: MvccSnapshot[applied={T|T < 7334203072273559552}]
I20260928 06:28:41.946394 28011 mvcc.cc:205] Tried to move back new op lower bound from 7334203072273559552 to 7334203072224030720. Current Snapshot: MvccSnapshot[applied={T|T < 7334203072273559552 or (T in {7334203072273559552})}]
I20260928 06:28:41.946506 28012 mvcc.cc:205] Tried to move back new op lower bound from 7334203072273559552 to 7334203072224030720. Current Snapshot: MvccSnapshot[applied={T|T < 7334203072273559552 or (T in {7334203072273559552})}]
I20260928 06:28:41.947660 27524 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51282:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:28:41.947765 27524 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51282:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:28:41.948477 27524 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d71bb86412684ec1a52f31347cd76025: Sending DeleteTablet for 3 replicas of tablet e954cbe48b6048ac93fa425d0175fda4
I20260928 06:28:41.948855 27648 tablet_service.cc:1514] Processing DeleteTablet for tablet e954cbe48b6048ac93fa425d0175fda4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:28:41 UTC) from {username='kudu', principal='kudu/127.22.225.190@KRBTEST.COM'} at 127.0.0.1:44868
I20260928 06:28:41.948866 27781 tablet_service.cc:1514] Processing DeleteTablet for tablet e954cbe48b6048ac93fa425d0175fda4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:28:41 UTC) from {username='kudu', principal='kudu/127.22.225.190@KRBTEST.COM'} at 127.0.0.1:45766
I20260928 06:28:41.949075 28020 tablet_replica.cc:330] T e954cbe48b6048ac93fa425d0175fda4 P 4ed4756c1d924686834e7b22938f9329: stopping tablet replica
I20260928 06:28:41.949172 28020 raft_consensus.cc:2229] T e954cbe48b6048ac93fa425d0175fda4 P 4ed4756c1d924686834e7b22938f9329 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:28:41.949226 27914 tablet_service.cc:1514] Processing DeleteTablet for tablet e954cbe48b6048ac93fa425d0175fda4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:28:41 UTC) from {username='kudu', principal='kudu/127.22.225.190@KRBTEST.COM'} at 127.0.0.1:37438
I20260928 06:28:41.949271 28020 raft_consensus.cc:2258] T e954cbe48b6048ac93fa425d0175fda4 P 4ed4756c1d924686834e7b22938f9329 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:28:41.949551 28019 tablet_replica.cc:330] T e954cbe48b6048ac93fa425d0175fda4 P 004a05d797a546b1aff1780bca033841: stopping tablet replica
I20260928 06:28:41.949556 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 27586
I20260928 06:28:41.949656 28020 ts_tablet_manager.cc:1905] T e954cbe48b6048ac93fa425d0175fda4 P 4ed4756c1d924686834e7b22938f9329: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:28:41.950333 28021 tablet_replica.cc:330] T e954cbe48b6048ac93fa425d0175fda4 P 9c1c6f41f79c4ad3bf26379397d12f9a: stopping tablet replica
I20260928 06:28:41.950420 28021 raft_consensus.cc:2229] T e954cbe48b6048ac93fa425d0175fda4 P 9c1c6f41f79c4ad3bf26379397d12f9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:28:41.950507 28021 raft_consensus.cc:2258] T e954cbe48b6048ac93fa425d0175fda4 P 9c1c6f41f79c4ad3bf26379397d12f9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:28:41.950901 28021 ts_tablet_manager.cc:1905] T e954cbe48b6048ac93fa425d0175fda4 P 9c1c6f41f79c4ad3bf26379397d12f9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:28:41.951354 28020 ts_tablet_manager.cc:1918] T e954cbe48b6048ac93fa425d0175fda4 P 4ed4756c1d924686834e7b22938f9329: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:28:41.951411 28020 log.cc:1196] T e954cbe48b6048ac93fa425d0175fda4 P 4ed4756c1d924686834e7b22938f9329: Deleting WAL directory at /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/minicluster-data/ts-1/wal/wals/e954cbe48b6048ac93fa425d0175fda4
I20260928 06:28:41.951673 28020 ts_tablet_manager.cc:1939] T e954cbe48b6048ac93fa425d0175fda4 P 4ed4756c1d924686834e7b22938f9329: Deleting consensus metadata
I20260928 06:28:41.952023 27509 catalog_manager.cc:4841] TS 4ed4756c1d924686834e7b22938f9329 (127.22.225.130:44137): tablet e954cbe48b6048ac93fa425d0175fda4 (table test-table [id=337a0d2e756a48a68cbf9f743a452037]) successfully deleted
I20260928 06:28:41.952811 28021 ts_tablet_manager.cc:1918] T e954cbe48b6048ac93fa425d0175fda4 P 9c1c6f41f79c4ad3bf26379397d12f9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:28:41.952868 28021 log.cc:1196] T e954cbe48b6048ac93fa425d0175fda4 P 9c1c6f41f79c4ad3bf26379397d12f9a: Deleting WAL directory at /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576899143097-23430-0/minicluster-data/ts-2/wal/wals/e954cbe48b6048ac93fa425d0175fda4
I20260928 06:28:41.953119 28021 ts_tablet_manager.cc:1939] T e954cbe48b6048ac93fa425d0175fda4 P 9c1c6f41f79c4ad3bf26379397d12f9a: Deleting consensus metadata
I20260928 06:28:41.953459 27509 catalog_manager.cc:4841] TS 9c1c6f41f79c4ad3bf26379397d12f9a (127.22.225.131:39345): tablet e954cbe48b6048ac93fa425d0175fda4 (table test-table [id=337a0d2e756a48a68cbf9f743a452037]) successfully deleted
W20260928 06:28:41.956969 27511 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.225.129:35071 (error 108)
I20260928 06:28:41.957459 27994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:41.957333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.129:35071 (local address 127.0.0.1:44876)
0928 06:28:41.957387 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:41.957427 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.129:35071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 06:28:41.957446 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 27719
W20260928 06:28:41.960173 27511 catalog_manager.cc:4584] TS 004a05d797a546b1aff1780bca033841 (127.22.225.129:35071): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e954cbe48b6048ac93fa425d0175fda4: Network error: Client connection negotiation failed: client connection to 127.22.225.129:35071: connect: Connection refused (error 111)
I20260928 06:28:41.963954 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 27852
I20260928 06:28:41.969817 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 27493
2026-09-28T06:28:41Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2013 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T06:28:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:28:41Z Disabled control of system clock
I20260928 06:28:41.998076 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576899143097-23430-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576899143097-23430-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576899143097-23430-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:42415
--webserver_interface=127.22.225.190
--webserver_port=0
--builtin_ntp_servers=127.22.225.148:41867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.225.190:42415
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 06:28:42.076264 28026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:28:42.076490 28026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:28:42.076526 28026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:28:42.077594 28026 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:28:42.077652 28026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:28:42.077673 28026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:28:42.077703 28026 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:28:42.077726 28026 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:28:42.079041 28026 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:41867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576899143097-23430-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576899143097-23430-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.225.190:42415
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:42415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576899143097-23430-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:28:42.079309 28026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:28:42.079571 28026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:28:42.084259 28031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:42.084329 28032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:42.084273 28034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:42.084692 28026 server_base.cc:1034] running on GCE node
I20260928 06:28:42.084959 28026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:28:42.085253 28026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:28:42.086404 28026 hybrid_clock.cc:648] HybridClock initialized: now 1790576922086387 us; error 34 us; skew 500 ppm
I20260928 06:28:42.087710 28026 webserver.cc:460] Webserver started at http://127.22.225.190:35043/ using document root <none> and password file <none>
I20260928 06:28:42.087965 28026 fs_manager.cc:359] Metadata directory not provided
I20260928 06:28:42.088012 28026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:28:42.088167 28026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:28:42.089318 28026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576899143097-23430-0/minicluster-data/master-0/data/instance:
uuid: "849fa2372e2a4ed5b5cda574f30cb164"
format_stamp: "Formatted at 2026-09-28 06:28:42 on dist-test-slave-d59m"
server_key: "cda77bb72e946d7655f029ce3e466812"
server_key_iv: "ddb123f8ead5bea7b8a520f71dd19e7d"
server_key_version: "encryptionkey@0"
I20260928 06:28:42.089664 28026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576899143097-23430-0/minicluster-data/master-0/wal/instance:
uuid: "849fa2372e2a4ed5b5cda574f30cb164"
format_stamp: "Formatted at 2026-09-28 06:28:42 on dist-test-slave-d59m"
server_key: "cda77bb72e946d7655f029ce3e466812"
server_key_iv: "ddb123f8ead5bea7b8a520f71dd19e7d"
server_key_version: "encryptionkey@0"
I20260928 06:28:42.090953 28026 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:28:42.091704 28040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:42.091977 28026 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:28:42.092095 28026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576899143097-23430-0/minicluster-data/master-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576899143097-23430-0/minicluster-data/master-0/wal
uuid: "849fa2372e2a4ed5b5cda574f30cb164"
format_stamp: "Formatted at 2026-09-28 06:28:42 on dist-test-slave-d59m"
server_key: "cda77bb72e946d7655f029ce3e466812"
server_key_iv: "ddb123f8ead5bea7b8a520f71dd19e7d"
server_key_version: "encryptionkey@0"
I20260928 06:28:42.092190 28026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576899143097-23430-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576899143097-23430-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576899143097-23430-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:28:42.096694 28026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:28:42.096942 28026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:28:42.097062 28026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:28:42.101490 28026 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.190:42415
I20260928 06:28:42.101550 28093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.190:42415 every 8 connection(s)
I20260928 06:28:42.101915 28026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
I20260928 06:28:42.102595 28094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:42.105429 28094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 849fa2372e2a4ed5b5cda574f30cb164: Bootstrap starting.
I20260928 06:28:42.106029 28094 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 849fa2372e2a4ed5b5cda574f30cb164: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:42.106317 28094 log.cc:824] T 00000000000000000000000000000000 P 849fa2372e2a4ed5b5cda574f30cb164: Log is configured to *not* fsync() on all Append() calls
I20260928 06:28:42.106985 28094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 849fa2372e2a4ed5b5cda574f30cb164: No bootstrap required, opened a new log
I20260928 06:28:42.108552 28094 raft_consensus.cc:348] T 00000000000000000000000000000000 P 849fa2372e2a4ed5b5cda574f30cb164 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "849fa2372e2a4ed5b5cda574f30cb164" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 42415 } }
I20260928 06:28:42.108772 28094 raft_consensus.cc:374] T 00000000000000000000000000000000 P 849fa2372e2a4ed5b5cda574f30cb164 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:42.108840 28094 raft_consensus.cc:729] T 00000000000000000000000000000000 P 849fa2372e2a4ed5b5cda574f30cb164 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 849fa2372e2a4ed5b5cda574f30cb164, State: Initialized, Role: FOLLOWER
I20260928 06:28:42.108989 28094 consensus_queue.cc:260] T 00000000000000000000000000000000 P 849fa2372e2a4ed5b5cda574f30cb164 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "849fa2372e2a4ed5b5cda574f30cb164" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 42415 } }
I20260928 06:28:42.109071 28094 raft_consensus.cc:388] T 00000000000000000000000000000000 P 849fa2372e2a4ed5b5cda574f30cb164 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:28:42.109115 28094 raft_consensus.cc:482] T 00000000000000000000000000000000 P 849fa2372e2a4ed5b5cda574f30cb164 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:28:42.109174 28094 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 849fa2372e2a4ed5b5cda574f30cb164 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:42.109678 28094 raft_consensus.cc:504] T 00000000000000000000000000000000 P 849fa2372e2a4ed5b5cda574f30cb164 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "849fa2372e2a4ed5b5cda574f30cb164" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 42415 } }
I20260928 06:28:42.109783 28094 leader_election.cc:302] T 00000000000000000000000000000000 P 849fa2372e2a4ed5b5cda574f30cb164 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 849fa2372e2a4ed5b5cda574f30cb164; no voters:
I20260928 06:28:42.109963 28094 leader_election.cc:288] T 00000000000000000000000000000000 P 849fa2372e2a4ed5b5cda574f30cb164 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:28:42.109997 28097 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 849fa2372e2a4ed5b5cda574f30cb164 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:28:42.110116 28097 raft_consensus.cc:686] T 00000000000000000000000000000000 P 849fa2372e2a4ed5b5cda574f30cb164 [term 1 LEADER]: Becoming Leader. State: Replica: 849fa2372e2a4ed5b5cda574f30cb164, State: Running, Role: LEADER
I20260928 06:28:42.110208 28094 sys_catalog.cc:564] T 00000000000000000000000000000000 P 849fa2372e2a4ed5b5cda574f30cb164 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:28:42.110200 28097 consensus_queue.cc:237] T 00000000000000000000000000000000 P 849fa2372e2a4ed5b5cda574f30cb164 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "849fa2372e2a4ed5b5cda574f30cb164" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 42415 } }
I20260928 06:28:42.110499 28097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 849fa2372e2a4ed5b5cda574f30cb164 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 849fa2372e2a4ed5b5cda574f30cb164. Latest consensus state: current_term: 1 leader_uuid: "849fa2372e2a4ed5b5cda574f30cb164" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "849fa2372e2a4ed5b5cda574f30cb164" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 42415 } } }
I20260928 06:28:42.110574 28097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 849fa2372e2a4ed5b5cda574f30cb164 [sys.catalog]: This master's current role is: LEADER
I20260928 06:28:42.110508 28098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 849fa2372e2a4ed5b5cda574f30cb164 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "849fa2372e2a4ed5b5cda574f30cb164" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "849fa2372e2a4ed5b5cda574f30cb164" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 42415 } } }
I20260928 06:28:42.110733 28098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 849fa2372e2a4ed5b5cda574f30cb164 [sys.catalog]: This master's current role is: LEADER
I20260928 06:28:42.111047 28102 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:28:42.111548 28102 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:28:42.112159 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 28026
I20260928 06:28:42.112267 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576899143097-23430-0/minicluster-data/master-0/wal/instance
I20260928 06:28:42.112447 23430 external_mini_cluster.cc:1428] Setting key e78d519d04be475c7fda03e4146c4238
I20260928 06:28:42.114643 28102 catalog_manager.cc:1347] Generated new cluster ID: 123668dac76a4d8d8c0f9ebceac2609c
I20260928 06:28:42.114702 28102 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:28:42.115073 28116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:42.113096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47406 (local address 127.22.225.190:42415)
0928 06:28:42.113275 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:42.113285 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:42.113506 (+ 221us) server_negotiation.cc:407] Connection header received
0928 06:28:42.113630 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:42.113636 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:42.113670 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:42.113816 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:42.114596 (+ 780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:42.114600 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:42.114601 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:28:42.114606 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:42.114651 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:28:42.114667 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:42.114708 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:42.114841 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 06:28:42.114899 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":84,"threads_started":1}
I20260928 06:28:42.120920 28102 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:28:42.121098 28102 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:28:42.127061 28102 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 849fa2372e2a4ed5b5cda574f30cb164: Generated new TSK 0
I20260928 06:28:42.127250 28102 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:28:42.134482 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576899143097-23430-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576899143097-23430-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576899143097-23430-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.129:0
--local_ip_for_outbound_sockets=127.22.225.129
--webserver_interface=127.22.225.129
--webserver_port=0
--tserver_master_addrs=127.22.225.190:42415
--builtin_ntp_servers=127.22.225.148:41867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 06:28:42.253223 28118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:28:42.253525 28118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:28:42.253557 28118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:28:42.255368 28118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:28:42.255420 28118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:28:42.255460 28118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.129
I20260928 06:28:42.257436 28118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:41867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576899143097-23430-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576899143097-23430-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.225.129
--webserver_port=0
--tserver_master_addrs=127.22.225.190:42415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.129
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576899143097-23430-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:28:42.257706 28118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:28:42.258131 28118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:28:42.263469 28123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:42.263480 28126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:42.263654 28118 server_base.cc:1034] running on GCE node
W20260928 06:28:42.263793 28124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:42.264029 28118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:28:42.264285 28118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:28:42.265448 28118 hybrid_clock.cc:648] HybridClock initialized: now 1790576922265430 us; error 33 us; skew 500 ppm
I20260928 06:28:42.266911 28118 webserver.cc:460] Webserver started at http://127.22.225.129:33239/ using document root <none> and password file <none>
I20260928 06:28:42.267182 28118 fs_manager.cc:359] Metadata directory not provided
I20260928 06:28:42.267246 28118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:28:42.267377 28118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:28:42.268743 28118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576899143097-23430-0/minicluster-data/ts-0/data/instance:
uuid: "282544e067434199ad84e68c3d542d64"
format_stamp: "Formatted at 2026-09-28 06:28:42 on dist-test-slave-d59m"
server_key: "b6f80ac9554b1f8340d723556238e589"
server_key_iv: "337ad95cb7d1395ec79b9a9bf7cd899b"
server_key_version: "encryptionkey@0"
I20260928 06:28:42.269124 28118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576899143097-23430-0/minicluster-data/ts-0/wal/instance:
uuid: "282544e067434199ad84e68c3d542d64"
format_stamp: "Formatted at 2026-09-28 06:28:42 on dist-test-slave-d59m"
server_key: "b6f80ac9554b1f8340d723556238e589"
server_key_iv: "337ad95cb7d1395ec79b9a9bf7cd899b"
server_key_version: "encryptionkey@0"
I20260928 06:28:42.270642 28118 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 06:28:42.271474 28132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:42.271672 28118 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:28:42.271754 28118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576899143097-23430-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576899143097-23430-0/minicluster-data/ts-0/wal
uuid: "282544e067434199ad84e68c3d542d64"
format_stamp: "Formatted at 2026-09-28 06:28:42 on dist-test-slave-d59m"
server_key: "b6f80ac9554b1f8340d723556238e589"
server_key_iv: "337ad95cb7d1395ec79b9a9bf7cd899b"
server_key_version: "encryptionkey@0"
I20260928 06:28:42.271838 28118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576899143097-23430-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576899143097-23430-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576899143097-23430-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:28:42.276322 28118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:28:42.276599 28118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:28:42.276718 28118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:28:42.276921 28118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:28:42.277208 28118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:28:42.277244 28118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:42.277269 28118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:28:42.277287 28118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:42.283833 28118 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.129:34599
I20260928 06:28:42.283960 28245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.129:34599 every 8 connection(s)
I20260928 06:28:42.284193 28118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
I20260928 06:28:42.287281 28116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:42.284947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:43591 (local address 127.22.225.190:42415)
0928 06:28:42.284996 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:42.284998 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:42.285298 (+ 300us) server_negotiation.cc:407] Connection header received
0928 06:28:42.285794 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:42.285795 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:42.285804 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:42.285909 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:42.287070 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:42.287071 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:42.287071 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:28:42.287075 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:42.287089 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:28:42.287093 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:42.287153 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:42.287202 (+ 49us) server_negotiation.cc:299] Negotiation successful
0928 06:28:42.287216 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 06:28:42.287506 28248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:42.284935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:42415 (local address 127.22.225.129:43591)
0928 06:28:42.285141 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:42.285191 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:42.285699 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:42.285967 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:42.285970 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:42.285978 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:42.286922 (+ 944us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:28:42.286923 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:42.286950 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:28:42.286976 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:42.287150 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:42.287153 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:42.287155 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:28:42.287200 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 06:28:42.287243 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":82,"threads_started":1}
I20260928 06:28:42.287930 28246 heartbeater.cc:347] Connected to a master server at 127.22.225.190:42415
I20260928 06:28:42.288070 28246 heartbeater.cc:464] Registering TS with master...
I20260928 06:28:42.288218 28246 heartbeater.cc:511] Master 127.22.225.190:42415 requested a full tablet report, sending...
I20260928 06:28:42.288753 28057 ts_manager.cc:194] Registered new tserver with Master: 282544e067434199ad84e68c3d542d64 (127.22.225.129:34599)
I20260928 06:28:42.294155 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 28118
I20260928 06:28:42.294238 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576899143097-23430-0/minicluster-data/ts-0/wal/instance
I20260928 06:28:42.294418 23430 external_mini_cluster.cc:1428] Setting key 9cd220e37f6135a96afd097f4812cfa3
I20260928 06:28:42.295302 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576899143097-23430-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576899143097-23430-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576899143097-23430-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.130:0
--local_ip_for_outbound_sockets=127.22.225.130
--webserver_interface=127.22.225.130
--webserver_port=0
--tserver_master_addrs=127.22.225.190:42415
--builtin_ntp_servers=127.22.225.148:41867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 06:28:42.377135 28249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:28:42.377334 28249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:28:42.377362 28249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:28:42.378573 28249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:28:42.378612 28249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:28:42.378639 28249 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.130
I20260928 06:28:42.380067 28249 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:41867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576899143097-23430-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576899143097-23430-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.225.130
--webserver_port=0
--tserver_master_addrs=127.22.225.190:42415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28249
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.130
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576899143097-23430-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:28:42.380316 28249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:28:42.380661 28249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:28:42.385529 28255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:42.385547 28254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:42.385581 28257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:42.385845 28249 server_base.cc:1034] running on GCE node
I20260928 06:28:42.386122 28249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:28:42.386401 28249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:28:42.387554 28249 hybrid_clock.cc:648] HybridClock initialized: now 1790576922387544 us; error 26 us; skew 500 ppm
I20260928 06:28:42.388933 28249 webserver.cc:460] Webserver started at http://127.22.225.130:36195/ using document root <none> and password file <none>
I20260928 06:28:42.389209 28249 fs_manager.cc:359] Metadata directory not provided
I20260928 06:28:42.389276 28249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:28:42.389405 28249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:28:42.390542 28249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576899143097-23430-0/minicluster-data/ts-1/data/instance:
uuid: "5fc5a7fb391a49859244a0df091764df"
format_stamp: "Formatted at 2026-09-28 06:28:42 on dist-test-slave-d59m"
server_key: "c69bd9e60ec511fbc3527f9f1bbd4dfc"
server_key_iv: "f4707ad73cf8d63fbec5cdf0bb434e9c"
server_key_version: "encryptionkey@0"
I20260928 06:28:42.390893 28249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576899143097-23430-0/minicluster-data/ts-1/wal/instance:
uuid: "5fc5a7fb391a49859244a0df091764df"
format_stamp: "Formatted at 2026-09-28 06:28:42 on dist-test-slave-d59m"
server_key: "c69bd9e60ec511fbc3527f9f1bbd4dfc"
server_key_iv: "f4707ad73cf8d63fbec5cdf0bb434e9c"
server_key_version: "encryptionkey@0"
I20260928 06:28:42.392237 28249 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:28:42.393060 28263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:42.393256 28249 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:28:42.393362 28249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576899143097-23430-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576899143097-23430-0/minicluster-data/ts-1/wal
uuid: "5fc5a7fb391a49859244a0df091764df"
format_stamp: "Formatted at 2026-09-28 06:28:42 on dist-test-slave-d59m"
server_key: "c69bd9e60ec511fbc3527f9f1bbd4dfc"
server_key_iv: "f4707ad73cf8d63fbec5cdf0bb434e9c"
server_key_version: "encryptionkey@0"
I20260928 06:28:42.393445 28249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576899143097-23430-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576899143097-23430-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576899143097-23430-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:28:42.398644 28249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:28:42.398962 28249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:28:42.399109 28249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:28:42.399348 28249 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:28:42.399716 28249 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:28:42.399767 28249 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:42.399797 28249 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:28:42.399821 28249 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:42.407300 28249 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.130:42501
I20260928 06:28:42.407437 28376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.130:42501 every 8 connection(s)
I20260928 06:28:42.407728 28249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
I20260928 06:28:42.410369 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 28249
I20260928 06:28:42.410485 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576899143097-23430-0/minicluster-data/ts-1/wal/instance
I20260928 06:28:42.410702 23430 external_mini_cluster.cc:1428] Setting key ecb1f3cc24ef3bd1e97855b5319767d6
I20260928 06:28:42.411803 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576899143097-23430-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576899143097-23430-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576899143097-23430-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.131:0
--local_ip_for_outbound_sockets=127.22.225.131
--webserver_interface=127.22.225.131
--webserver_port=0
--tserver_master_addrs=127.22.225.190:42415
--builtin_ntp_servers=127.22.225.148:41867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 06:28:42.414417 28116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:42.408616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:51523 (local address 127.22.225.190:42415)
0928 06:28:42.408686 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:42.408689 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:42.409018 (+ 329us) server_negotiation.cc:407] Connection header received
0928 06:28:42.409642 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:42.409643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:42.409651 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:42.409733 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:42.414091 (+ 4358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:42.414093 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:42.414093 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:28:42.414098 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:42.414120 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:28:42.414125 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:42.414203 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:42.414326 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 06:28:42.414342 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 06:28:42.414868 28379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:42.408568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:42415 (local address 127.22.225.130:51523)
0928 06:28:42.408799 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:42.408906 (+ 107us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:42.409515 (+ 609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:42.409805 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:42.409809 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:42.409819 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:42.413860 (+ 4041us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:28:42.413861 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:42.413909 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:28:42.413954 (+ 45us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:42.414232 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:42.414239 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:42.414243 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:28:42.414570 (+ 327us) client_negotiation.cc:240] Negotiation successful
0928 06:28:42.414643 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":89,"threads_started":1}
I20260928 06:28:42.415444 28377 heartbeater.cc:347] Connected to a master server at 127.22.225.190:42415
I20260928 06:28:42.415544 28377 heartbeater.cc:464] Registering TS with master...
I20260928 06:28:42.415701 28377 heartbeater.cc:511] Master 127.22.225.190:42415 requested a full tablet report, sending...
I20260928 06:28:42.416108 28057 ts_manager.cc:194] Registered new tserver with Master: 5fc5a7fb391a49859244a0df091764df (127.22.225.130:42501)
W20260928 06:28:42.510737 28380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:28:42.510998 28380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:28:42.511019 28380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:28:42.512246 28380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:28:42.512284 28380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:28:42.512310 28380 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.131
I20260928 06:28:42.513615 28380 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:41867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576899143097-23430-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576899143097-23430-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.225.131
--webserver_port=0
--tserver_master_addrs=127.22.225.190:42415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28380
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.131
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576899143097-23430-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:28:42.513800 28380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:28:42.514042 28380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:28:42.519094 28385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:42.519086 28388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:42.519114 28386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:42.519263 28380 server_base.cc:1034] running on GCE node
I20260928 06:28:42.519536 28380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:28:42.519793 28380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:28:42.520994 28380 hybrid_clock.cc:648] HybridClock initialized: now 1790576922520986 us; error 38 us; skew 500 ppm
I20260928 06:28:42.522291 28380 webserver.cc:460] Webserver started at http://127.22.225.131:46769/ using document root <none> and password file <none>
I20260928 06:28:42.522529 28380 fs_manager.cc:359] Metadata directory not provided
I20260928 06:28:42.522620 28380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:28:42.522789 28380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:28:42.523908 28380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576899143097-23430-0/minicluster-data/ts-2/data/instance:
uuid: "7fbd1aa844ca4e5e9a8e569671ea9f81"
format_stamp: "Formatted at 2026-09-28 06:28:42 on dist-test-slave-d59m"
server_key: "0df16d39df4b527cc3dd0a8c9d2769a1"
server_key_iv: "aab78a2fa59f000c63212e52e3c2dc2d"
server_key_version: "encryptionkey@0"
I20260928 06:28:42.524286 28380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576899143097-23430-0/minicluster-data/ts-2/wal/instance:
uuid: "7fbd1aa844ca4e5e9a8e569671ea9f81"
format_stamp: "Formatted at 2026-09-28 06:28:42 on dist-test-slave-d59m"
server_key: "0df16d39df4b527cc3dd0a8c9d2769a1"
server_key_iv: "aab78a2fa59f000c63212e52e3c2dc2d"
server_key_version: "encryptionkey@0"
I20260928 06:28:42.525681 28380 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:28:42.526437 28394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:42.526607 28380 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:28:42.526703 28380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576899143097-23430-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576899143097-23430-0/minicluster-data/ts-2/wal
uuid: "7fbd1aa844ca4e5e9a8e569671ea9f81"
format_stamp: "Formatted at 2026-09-28 06:28:42 on dist-test-slave-d59m"
server_key: "0df16d39df4b527cc3dd0a8c9d2769a1"
server_key_iv: "aab78a2fa59f000c63212e52e3c2dc2d"
server_key_version: "encryptionkey@0"
I20260928 06:28:42.526790 28380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576899143097-23430-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576899143097-23430-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576899143097-23430-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:28:42.530189 28380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:28:42.530452 28380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:28:42.530588 28380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:28:42.530805 28380 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:28:42.531106 28380 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:28:42.531157 28380 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:42.531194 28380 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:28:42.531221 28380 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:42.537035 28380 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.131:42289
I20260928 06:28:42.537091 28507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.131:42289 every 8 connection(s)
I20260928 06:28:42.537421 28380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
I20260928 06:28:42.537858 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 28380
I20260928 06:28:42.537936 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576899143097-23430-0/minicluster-data/ts-2/wal/instance
I20260928 06:28:42.538148 23430 external_mini_cluster.cc:1428] Setting key 27db4713f5617856e9f720a6b70d438b
I20260928 06:28:42.540493 28116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:42.538193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:50307 (local address 127.22.225.190:42415)
0928 06:28:42.538273 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:42.538276 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:42.538552 (+ 276us) server_negotiation.cc:407] Connection header received
0928 06:28:42.539202 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:42.539202 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:42.539207 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:42.539257 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:42.540280 (+ 1023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:42.540281 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:42.540281 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:28:42.540284 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:42.540296 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:28:42.540300 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:42.540343 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:42.540428 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 06:28:42.540440 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 06:28:42.540723 28510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:42.538174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:42415 (local address 127.22.225.131:50307)
0928 06:28:42.538407 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:42.538450 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:42.539112 (+ 662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:42.539308 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:42.539311 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:42.539320 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:42.540155 (+ 835us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:28:42.540156 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:42.540181 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:28:42.540206 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:42.540341 (+ 135us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:42.540343 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:42.540345 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:28:42.540414 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 06:28:42.540461 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":85,"threads_started":1}
I20260928 06:28:42.541224 28508 heartbeater.cc:347] Connected to a master server at 127.22.225.190:42415
I20260928 06:28:42.541320 28508 heartbeater.cc:464] Registering TS with master...
I20260928 06:28:42.541464 28508 heartbeater.cc:511] Master 127.22.225.190:42415 requested a full tablet report, sending...
I20260928 06:28:42.541841 28057 ts_manager.cc:194] Registered new tserver with Master: 7fbd1aa844ca4e5e9a8e569671ea9f81 (127.22.225.131:42289)
I20260928 06:28:42.549901 23430 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:28:42.552384 28116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:42.551088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47408 (local address 127.22.225.190:42415)
0928 06:28:42.551127 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:42.551129 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:42.551316 (+ 187us) server_negotiation.cc:407] Connection header received
0928 06:28:42.551412 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:42.551413 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:42.551436 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:42.551486 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:42.552158 (+ 672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:42.552160 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:42.552160 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:28:42.552164 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:42.552180 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:28:42.552184 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:42.552241 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:42.552291 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 06:28:42.552308 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 06:28:42.553936 28057 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47408:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:28:42.555491 28057 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:28:42.559751 28520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:42.557897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35488 (local address 127.22.225.131:42289)
0928 06:28:42.558080 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:42.558085 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:42.558193 (+ 108us) server_negotiation.cc:407] Connection header received
0928 06:28:42.558268 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:42.558271 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:42.558280 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:42.558363 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:42.559431 (+ 1068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:42.559438 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:42.559438 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:28:42.559442 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:42.559471 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:28:42.559503 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:42.559554 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:42.559649 (+ 95us) server_negotiation.cc:299] Negotiation successful
0928 06:28:42.559666 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":84,"threads_started":1}
I20260928 06:28:42.559751 28516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:42.557679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.131:42289 (local address 127.0.0.1:35488)
0928 06:28:42.558147 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:42.558155 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:42.558198 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:42.558655 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:42.558656 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:42.558658 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:42.559333 (+ 675us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:28:42.559334 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:42.559339 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:28:42.559349 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:42.559581 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:42.559582 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:42.559583 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 06:28:42.559635 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 06:28:42.559648 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":90,"threads_started":1}
I20260928 06:28:42.559805 28519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:42.557728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52266 (local address 127.22.225.129:34599)
0928 06:28:42.558110 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:42.558115 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:42.558126 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:28:42.558153 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:42.558156 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:42.558168 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:42.558247 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:42.559389 (+ 1142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:42.559399 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:42.559399 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:28:42.559404 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:42.559442 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:28:42.559470 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:42.559533 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:42.559698 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 06:28:42.559714 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":75,"threads_started":1}
I20260928 06:28:42.560032 28521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:42.557896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37598 (local address 127.22.225.130:42501)
0928 06:28:42.558643 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:42.558647 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:42.558654 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:28:42.558676 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:42.558677 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:42.558683 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:42.558765 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:42.559592 (+ 827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:42.559599 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:42.559600 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:28:42.559604 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:42.559636 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:28:42.559664 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:42.559717 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:42.559919 (+ 202us) server_negotiation.cc:299] Negotiation successful
0928 06:28:42.559937 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":703,"thread_start_us":104,"threads_started":1}
I20260928 06:28:42.559859 28517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:42.557678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.130:42501 (local address 127.0.0.1:37598)
0928 06:28:42.558012 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:42.558029 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:42.558076 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:42.558780 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:42.558781 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:42.558784 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:42.559497 (+ 713us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:28:42.559498 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:42.559506 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:28:42.559521 (+ 15us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:42.559741 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:42.559742 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:42.559742 (+ 0us) client_negotiation.cc:769] Sending connection context
0928 06:28:42.559784 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 06:28:42.559795 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":87,"threads_started":1}
I20260928 06:28:42.559937 28518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:42.557646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.129:34599 (local address 127.0.0.1:52266)
0928 06:28:42.557827 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:42.557867 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:42.557923 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:42.558262 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:42.558268 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:42.558275 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:42.559234 (+ 959us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:28:42.559235 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:42.559248 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:28:42.559293 (+ 45us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:42.559529 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:42.559536 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:42.559539 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:28:42.559617 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 06:28:42.559633 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":69,"threads_started":1}
I20260928 06:28:42.561059 28442 tablet_service.cc:1467] Processing CreateTablet for tablet dc8f6e53bd4e47e9822793d8b8ba6bd5 (DEFAULT_TABLE table=test-table [id=eb5e54b4c513425ba8595bed60780c50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:28:42.561305 28180 tablet_service.cc:1467] Processing CreateTablet for tablet dc8f6e53bd4e47e9822793d8b8ba6bd5 (DEFAULT_TABLE table=test-table [id=eb5e54b4c513425ba8595bed60780c50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:28:42.561408 28442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc8f6e53bd4e47e9822793d8b8ba6bd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:42.561636 28180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc8f6e53bd4e47e9822793d8b8ba6bd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:42.564082 28522 tablet_bootstrap.cc:492] T dc8f6e53bd4e47e9822793d8b8ba6bd5 P 7fbd1aa844ca4e5e9a8e569671ea9f81: Bootstrap starting.
I20260928 06:28:42.564513 28311 tablet_service.cc:1467] Processing CreateTablet for tablet dc8f6e53bd4e47e9822793d8b8ba6bd5 (DEFAULT_TABLE table=test-table [id=eb5e54b4c513425ba8595bed60780c50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:28:42.564858 28311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc8f6e53bd4e47e9822793d8b8ba6bd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:42.564980 28522 tablet_bootstrap.cc:654] T dc8f6e53bd4e47e9822793d8b8ba6bd5 P 7fbd1aa844ca4e5e9a8e569671ea9f81: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:42.565339 28522 log.cc:824] T dc8f6e53bd4e47e9822793d8b8ba6bd5 P 7fbd1aa844ca4e5e9a8e569671ea9f81: Log is configured to *not* fsync() on all Append() calls
I20260928 06:28:42.566008 28523 tablet_bootstrap.cc:492] T dc8f6e53bd4e47e9822793d8b8ba6bd5 P 282544e067434199ad84e68c3d542d64: Bootstrap starting.
I20260928 06:28:42.566262 28522 tablet_bootstrap.cc:492] T dc8f6e53bd4e47e9822793d8b8ba6bd5 P 7fbd1aa844ca4e5e9a8e569671ea9f81: No bootstrap required, opened a new log
I20260928 06:28:42.566325 28522 ts_tablet_manager.cc:1397] T dc8f6e53bd4e47e9822793d8b8ba6bd5 P 7fbd1aa844ca4e5e9a8e569671ea9f81: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:28:42.566659 28523 tablet_bootstrap.cc:654] T dc8f6e53bd4e47e9822793d8b8ba6bd5 P 282544e067434199ad84e68c3d542d64: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:42.566958 28523 log.cc:824] T dc8f6e53bd4e47e9822793d8b8ba6bd5 P 282544e067434199ad84e68c3d542d64: Log is configured to *not* fsync() on all Append() calls
I20260928 06:28:42.567705 28523 tablet_bootstrap.cc:492] T dc8f6e53bd4e47e9822793d8b8ba6bd5 P 282544e067434199ad84e68c3d542d64: No bootstrap required, opened a new log
I20260928 06:28:42.567760 28523 ts_tablet_manager.cc:1397] T dc8f6e53bd4e47e9822793d8b8ba6bd5 P 282544e067434199ad84e68c3d542d64: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:28:42.568472 28526 tablet_bootstrap.cc:492] T dc8f6e53bd4e47e9822793d8b8ba6bd5 P 5fc5a7fb391a49859244a0df091764df: Bootstrap starting.
I20260928 06:28:42.568863 28522 raft_consensus.cc:348] T dc8f6e53bd4e47e9822793d8b8ba6bd5 P 7fbd1aa844ca4e5e9a8e569671ea9f81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "282544e067434199ad84e68c3d542d64" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34599 } } peers { permanent_uuid: "7fbd1aa844ca4e5e9a8e569671ea9f81" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 42289 } } peers { permanent_uuid: "5fc5a7fb391a49859244a0df091764df" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 42501 } }
I20260928 06:28:42.569013 28522 raft_consensus.cc:374] T dc8f6e53bd4e47e9822793d8b8ba6bd5 P 7fbd1aa844ca4e5e9a8e569671ea9f81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:42.569043 28522 raft_consensus.cc:729] T dc8f6e53bd4e47e9822793d8b8ba6bd5 P 7fbd1aa844ca4e5e9a8e569671ea9f81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fbd1aa844ca4e5e9a8e569671ea9f81, State: Initialized, Role: FOLLOWER
I20260928 06:28:42.569103 28526 tablet_bootstrap.cc:654] T dc8f6e53bd4e47e9822793d8b8ba6bd5 P 5fc5a7fb391a49859244a0df091764df: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:42.569388 28526 log.cc:824] T dc8f6e53bd4e47e9822793d8b8ba6bd5 P 5fc5a7fb391a49859244a0df091764df: Log is configured to *not* fsync() on all Append() calls
I20260928 06:28:42.569566 28522 consensus_queue.cc:260] T dc8f6e53bd4e47e9822793d8b8ba6bd5 P 7fbd1aa844ca4e5e9a8e569671ea9f81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "282544e067434199ad84e68c3d542d64" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34599 } } peers { permanent_uuid: "7fbd1aa844ca4e5e9a8e569671ea9f81" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 42289 } } peers { permanent_uuid: "5fc5a7fb391a49859244a0df091764df" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 42501 } }
I20260928 06:28:42.569902 28522 ts_tablet_manager.cc:1428] T dc8f6e53bd4e47e9822793d8b8ba6bd5 P 7fbd1aa844ca4e5e9a8e569671ea9f81: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260928 06:28:42.570166 28526 tablet_bootstrap.cc:492] T dc8f6e53bd4e47e9822793d8b8ba6bd5 P 5fc5a7fb391a49859244a0df091764df: No bootstrap required, opened a new log
I20260928 06:28:42.570235 28526 ts_tablet_manager.cc:1397] T dc8f6e53bd4e47e9822793d8b8ba6bd5 P 5fc5a7fb391a49859244a0df091764df: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:28:42.570497 28523 raft_consensus.cc:348] T dc8f6e53bd4e47e9822793d8b8ba6bd5 P 282544e067434199ad84e68c3d542d64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "282544e067434199ad84e68c3d542d64" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34599 } } peers { permanent_uuid: "7fbd1aa844ca4e5e9a8e569671ea9f81" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 42289 } } peers { permanent_uuid: "5fc5a7fb391a49859244a0df091764df" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 42501 } }
I20260928 06:28:42.570665 28523 raft_consensus.cc:374] T dc8f6e53bd4e47e9822793d8b8ba6bd5 P 282544e067434199ad84e68c3d542d64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:42.570706 28523 raft_consensus.cc:729] T dc8f6e53bd4e47e9822793d8b8ba6bd5 P 282544e067434199ad84e68c3d542d64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 282544e067434199ad84e68c3d542d64, State: Initialized, Role: FOLLOWER
I20260928 06:28:42.570819 28523 consensus_queue.cc:260] T dc8f6e53bd4e47e9822793d8b8ba6bd5 P 282544e067434199ad84e68c3d542d64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "282544e067434199ad84e68c3d542d64" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34599 } } peers { permanent_uuid: "7fbd1aa844ca4e5e9a8e569671ea9f81" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 42289 } } peers { permanent_uuid: "5fc5a7fb391a49859244a0df091764df" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 42501 } }
I20260928 06:28:42.571053 28523 ts_tablet_manager.cc:1428] T dc8f6e53bd4e47e9822793d8b8ba6bd5 P 282544e067434199ad84e68c3d542d64: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 06:28:42.571180 28246 heartbeater.cc:503] Master 127.22.225.190:42415 was elected leader, sending a full tablet report...
I20260928 06:28:42.571257 28508 heartbeater.cc:503] Master 127.22.225.190:42415 was elected leader, sending a full tablet report...
I20260928 06:28:42.572837 28526 raft_consensus.cc:348] T dc8f6e53bd4e47e9822793d8b8ba6bd5 P 5fc5a7fb391a49859244a0df091764df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "282544e067434199ad84e68c3d542d64" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34599 } } peers { permanent_uuid: "7fbd1aa844ca4e5e9a8e569671ea9f81" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 42289 } } peers { permanent_uuid: "5fc5a7fb391a49859244a0df091764df" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 42501 } }
I20260928 06:28:42.572962 28526 raft_consensus.cc:374] T dc8f6e53bd4e47e9822793d8b8ba6bd5 P 5fc5a7fb391a49859244a0df091764df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:42.572984 28526 raft_consensus.cc:729] T dc8f6e53bd4e47e9822793d8b8ba6bd5 P 5fc5a7fb391a49859244a0df091764df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fc5a7fb391a49859244a0df091764df, State: Initialized, Role: FOLLOWER
I20260928 06:28:42.573122 28526 consensus_queue.cc:260] T dc8f6e53bd4e47e9822793d8b8ba6bd5 P 5fc5a7fb391a49859244a0df091764df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "282544e067434199ad84e68c3d542d64" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34599 } } peers { permanent_uuid: "7fbd1aa844ca4e5e9a8e569671ea9f81" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 42289 } } peers { permanent_uuid: "5fc5a7fb391a49859244a0df091764df" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 42501 } }
I20260928 06:28:42.573346 28526 ts_tablet_manager.cc:1428] T dc8f6e53bd4e47e9822793d8b8ba6bd5 P 5fc5a7fb391a49859244a0df091764df: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260928 06:28:42.573437 28377 heartbeater.cc:503] Master 127.22.225.190:42415 was elected leader, sending a full tablet report...
I20260928 06:28:42.595605 28528 raft_consensus.cc:482] T dc8f6e53bd4e47e9822793d8b8ba6bd5 P 7fbd1aa844ca4e5e9a8e569671ea9f81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:28:42.595728 28528 raft_consensus.cc:504] T dc8f6e53bd4e47e9822793d8b8ba6bd5 P 7fbd1aa844ca4e5e9a8e569671ea9f81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "282544e067434199ad84e68c3d542d64" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34599 } } peers { permanent_uuid: "7fbd1aa844ca4e5e9a8e569671ea9f81" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 42289 } } peers { permanent_uuid: "5fc5a7fb391a49859244a0df091764df" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 42501 } }
I20260928 06:28:42.596201 28528 leader_election.cc:288] T dc8f6e53bd4e47e9822793d8b8ba6bd5 P 7fbd1aa844ca4e5e9a8e569671ea9f81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 282544e067434199ad84e68c3d542d64 (127.22.225.129:34599), 5fc5a7fb391a49859244a0df091764df (127.22.225.130:42501)
I20260928 06:28:42.597635 28510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:42.596359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.130:42501 (local address 127.22.225.131:51341)
0928 06:28:42.596423 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:42.596435 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:42.596473 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:42.596788 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:42.596789 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:42.596792 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:42.597316 (+ 524us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:28:42.597317 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:42.597324 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:28:42.597338 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:42.597527 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:42.597528 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:42.597528 (+ 0us) client_negotiation.cc:769] Sending connection context
0928 06:28:42.597555 (+ 27us) client_negotiation.cc:240] Negotiation successful
0928 06:28:42.597567 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 06:28:42.597889 28519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:42.596555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:53691 (local address 127.22.225.129:34599)
0928 06:28:42.596616 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:42.596618 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:42.596683 (+ 65us) server_negotiation.cc:407] Connection header received
0928 06:28:42.596841 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:42.596842 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:42.596847 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:42.596919 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:42.597621 (+ 702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:42.597623 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:42.597623 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:28:42.597628 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:42.597641 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:28:42.597644 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:42.597709 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:42.597810 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 06:28:42.597824 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 06:28:42.597905 28531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:42.596361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.129:34599 (local address 127.22.225.131:53691)
0928 06:28:42.596612 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:42.596624 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:42.596691 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:42.596923 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:42.596924 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:42.596927 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:42.597486 (+ 559us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:28:42.597487 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:42.597495 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:28:42.597508 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:42.597730 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:42.597731 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:42.597731 (+ 0us) client_negotiation.cc:769] Sending connection context
0928 06:28:42.597804 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 06:28:42.597819 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":12,"thread_start_us":105,"threads_started":1}
I20260928 06:28:42.598209 28521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:42.596416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:51341 (local address 127.22.225.130:42501)
0928 06:28:42.596476 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:42.596478 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:42.596513 (+ 35us) server_negotiation.cc:407] Connection header received
0928 06:28:42.596541 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:42.596542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:42.596548 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:42.596777 (+ 229us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:42.597426 (+ 649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:42.597428 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:42.597428 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:28:42.597432 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:42.597448 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:28:42.597453 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:42.597908 (+ 455us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:42.597929 (+ 21us) server_negotiation.cc:299] Negotiation successful
0928 06:28:42.598142 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 06:28:42.598241 28200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc8f6e53bd4e47e9822793d8b8ba6bd5" candidate_uuid: "7fbd1aa844ca4e5e9a8e569671ea9f81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "282544e067434199ad84e68c3d542d64" is_pre_election: true
I20260928 06:28:42.598418 28200 raft_consensus.cc:2454] T dc8f6e53bd4e47e9822793d8b8ba6bd5 P 282544e067434199ad84e68c3d542d64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fbd1aa844ca4e5e9a8e569671ea9f81 in term 0.
I20260928 06:28:42.598459 28331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc8f6e53bd4e47e9822793d8b8ba6bd5" candidate_uuid: "7fbd1aa844ca4e5e9a8e569671ea9f81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fc5a7fb391a49859244a0df091764df" is_pre_election: true
I20260928 06:28:42.598644 28396 leader_election.cc:302] T dc8f6e53bd4e47e9822793d8b8ba6bd5 P 7fbd1aa844ca4e5e9a8e569671ea9f81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 282544e067434199ad84e68c3d542d64, 7fbd1aa844ca4e5e9a8e569671ea9f81; no voters:
I20260928 06:28:42.598728 28331 raft_consensus.cc:2454] T dc8f6e53bd4e47e9822793d8b8ba6bd5 P 5fc5a7fb391a49859244a0df091764df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fbd1aa844ca4e5e9a8e569671ea9f81 in term 0.
I20260928 06:28:42.598758 28528 raft_consensus.cc:2781] T dc8f6e53bd4e47e9822793d8b8ba6bd5 P 7fbd1aa844ca4e5e9a8e569671ea9f81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:28:42.598879 28528 raft_consensus.cc:482] T dc8f6e53bd4e47e9822793d8b8ba6bd5 P 7fbd1aa844ca4e5e9a8e569671ea9f81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:28:42.598913 28528 raft_consensus.cc:3037] T dc8f6e53bd4e47e9822793d8b8ba6bd5 P 7fbd1aa844ca4e5e9a8e569671ea9f81 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:42.599771 28528 raft_consensus.cc:504] T dc8f6e53bd4e47e9822793d8b8ba6bd5 P 7fbd1aa844ca4e5e9a8e569671ea9f81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "282544e067434199ad84e68c3d542d64" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34599 } } peers { permanent_uuid: "7fbd1aa844ca4e5e9a8e569671ea9f81" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 42289 } } peers { permanent_uuid: "5fc5a7fb391a49859244a0df091764df" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 42501 } }
I20260928 06:28:42.600014 28528 leader_election.cc:288] T dc8f6e53bd4e47e9822793d8b8ba6bd5 P 7fbd1aa844ca4e5e9a8e569671ea9f81 [CANDIDATE]: Term 1 election: Requested vote from peers 282544e067434199ad84e68c3d542d64 (127.22.225.129:34599), 5fc5a7fb391a49859244a0df091764df (127.22.225.130:42501)
I20260928 06:28:42.600041 28200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc8f6e53bd4e47e9822793d8b8ba6bd5" candidate_uuid: "7fbd1aa844ca4e5e9a8e569671ea9f81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "282544e067434199ad84e68c3d542d64"
I20260928 06:28:42.600066 28331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc8f6e53bd4e47e9822793d8b8ba6bd5" candidate_uuid: "7fbd1aa844ca4e5e9a8e569671ea9f81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fc5a7fb391a49859244a0df091764df"
I20260928 06:28:42.600111 28200 raft_consensus.cc:3037] T dc8f6e53bd4e47e9822793d8b8ba6bd5 P 282544e067434199ad84e68c3d542d64 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:42.600143 28331 raft_consensus.cc:3037] T dc8f6e53bd4e47e9822793d8b8ba6bd5 P 5fc5a7fb391a49859244a0df091764df [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:42.600927 28200 raft_consensus.cc:2454] T dc8f6e53bd4e47e9822793d8b8ba6bd5 P 282544e067434199ad84e68c3d542d64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fbd1aa844ca4e5e9a8e569671ea9f81 in term 1.
I20260928 06:28:42.601058 28331 raft_consensus.cc:2454] T dc8f6e53bd4e47e9822793d8b8ba6bd5 P 5fc5a7fb391a49859244a0df091764df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fbd1aa844ca4e5e9a8e569671ea9f81 in term 1.
I20260928 06:28:42.601239 28397 leader_election.cc:302] T dc8f6e53bd4e47e9822793d8b8ba6bd5 P 7fbd1aa844ca4e5e9a8e569671ea9f81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5fc5a7fb391a49859244a0df091764df, 7fbd1aa844ca4e5e9a8e569671ea9f81; no voters:
I20260928 06:28:42.601358 28528 raft_consensus.cc:2781] T dc8f6e53bd4e47e9822793d8b8ba6bd5 P 7fbd1aa844ca4e5e9a8e569671ea9f81 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:28:42.601549 28528 raft_consensus.cc:686] T dc8f6e53bd4e47e9822793d8b8ba6bd5 P 7fbd1aa844ca4e5e9a8e569671ea9f81 [term 1 LEADER]: Becoming Leader. State: Replica: 7fbd1aa844ca4e5e9a8e569671ea9f81, State: Running, Role: LEADER
I20260928 06:28:42.601652 28528 consensus_queue.cc:237] T dc8f6e53bd4e47e9822793d8b8ba6bd5 P 7fbd1aa844ca4e5e9a8e569671ea9f81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "282544e067434199ad84e68c3d542d64" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34599 } } peers { permanent_uuid: "7fbd1aa844ca4e5e9a8e569671ea9f81" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 42289 } } peers { permanent_uuid: "5fc5a7fb391a49859244a0df091764df" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 42501 } }
I20260928 06:28:42.602495 28056 catalog_manager.cc:5495] T dc8f6e53bd4e47e9822793d8b8ba6bd5 P 7fbd1aa844ca4e5e9a8e569671ea9f81 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7fbd1aa844ca4e5e9a8e569671ea9f81 (127.22.225.131). New cstate: current_term: 1 leader_uuid: "7fbd1aa844ca4e5e9a8e569671ea9f81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "282544e067434199ad84e68c3d542d64" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fbd1aa844ca4e5e9a8e569671ea9f81" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 42289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5fc5a7fb391a49859244a0df091764df" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 42501 } health_report { overall_health: UNKNOWN } } }
I20260928 06:28:42.612109 28520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:42.611034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35490 (local address 127.22.225.131:42289)
0928 06:28:42.611075 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:42.611077 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:42.611119 (+ 42us) server_negotiation.cc:407] Connection header received
0928 06:28:42.611205 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:42.611206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:42.611219 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:42.611271 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:42.611882 (+ 611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:42.611883 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:42.611883 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:28:42.611887 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:42.611903 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:28:42.611907 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:42.611965 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:42.612030 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 06:28:42.612042 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 06:28:42.614005 28200 raft_consensus.cc:1261] T dc8f6e53bd4e47e9822793d8b8ba6bd5 P 282544e067434199ad84e68c3d542d64 [term 1 FOLLOWER]: Refusing update from remote peer 7fbd1aa844ca4e5e9a8e569671ea9f81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:28:42.614048 28331 raft_consensus.cc:1261] T dc8f6e53bd4e47e9822793d8b8ba6bd5 P 5fc5a7fb391a49859244a0df091764df [term 1 FOLLOWER]: Refusing update from remote peer 7fbd1aa844ca4e5e9a8e569671ea9f81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:28:42.614377 28532 consensus_queue.cc:1035] T dc8f6e53bd4e47e9822793d8b8ba6bd5 P 7fbd1aa844ca4e5e9a8e569671ea9f81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fc5a7fb391a49859244a0df091764df" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:28:42.614490 28528 consensus_queue.cc:1035] T dc8f6e53bd4e47e9822793d8b8ba6bd5 P 7fbd1aa844ca4e5e9a8e569671ea9f81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "282544e067434199ad84e68c3d542d64" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:28:42.619421 28057 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47408:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:28:42.619534 28057 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47408:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:28:42.620227 28057 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 849fa2372e2a4ed5b5cda574f30cb164: Sending DeleteTablet for 3 replicas of tablet dc8f6e53bd4e47e9822793d8b8ba6bd5
I20260928 06:28:42.620568 28442 tablet_service.cc:1514] Processing DeleteTablet for tablet dc8f6e53bd4e47e9822793d8b8ba6bd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:28:42 UTC) from {username='slave'} at 127.0.0.1:35488
I20260928 06:28:42.620618 28180 tablet_service.cc:1514] Processing DeleteTablet for tablet dc8f6e53bd4e47e9822793d8b8ba6bd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:28:42 UTC) from {username='slave'} at 127.0.0.1:52266
I20260928 06:28:42.620652 28311 tablet_service.cc:1514] Processing DeleteTablet for tablet dc8f6e53bd4e47e9822793d8b8ba6bd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:28:42 UTC) from {username='slave'} at 127.0.0.1:37598
I20260928 06:28:42.620815 28546 tablet_replica.cc:330] T dc8f6e53bd4e47e9822793d8b8ba6bd5 P 7fbd1aa844ca4e5e9a8e569671ea9f81: stopping tablet replica
I20260928 06:28:42.620920 28546 raft_consensus.cc:2229] T dc8f6e53bd4e47e9822793d8b8ba6bd5 P 7fbd1aa844ca4e5e9a8e569671ea9f81 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:28:42.621008 28546 raft_consensus.cc:2258] T dc8f6e53bd4e47e9822793d8b8ba6bd5 P 7fbd1aa844ca4e5e9a8e569671ea9f81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:28:42.621074 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 28118
I20260928 06:28:42.621138 28545 tablet_replica.cc:330] T dc8f6e53bd4e47e9822793d8b8ba6bd5 P 282544e067434199ad84e68c3d542d64: stopping tablet replica
I20260928 06:28:42.621214 28545 raft_consensus.cc:2229] T dc8f6e53bd4e47e9822793d8b8ba6bd5 P 282544e067434199ad84e68c3d542d64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:28:42.621295 28545 raft_consensus.cc:2258] T dc8f6e53bd4e47e9822793d8b8ba6bd5 P 282544e067434199ad84e68c3d542d64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:28:42.621766 28546 ts_tablet_manager.cc:1905] T dc8f6e53bd4e47e9822793d8b8ba6bd5 P 7fbd1aa844ca4e5e9a8e569671ea9f81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:28:42.623802 28546 ts_tablet_manager.cc:1918] T dc8f6e53bd4e47e9822793d8b8ba6bd5 P 7fbd1aa844ca4e5e9a8e569671ea9f81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:28:42.623869 28546 log.cc:1196] T dc8f6e53bd4e47e9822793d8b8ba6bd5 P 7fbd1aa844ca4e5e9a8e569671ea9f81: Deleting WAL directory at /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576899143097-23430-0/minicluster-data/ts-2/wal/wals/dc8f6e53bd4e47e9822793d8b8ba6bd5
I20260928 06:28:42.624142 28546 ts_tablet_manager.cc:1939] T dc8f6e53bd4e47e9822793d8b8ba6bd5 P 7fbd1aa844ca4e5e9a8e569671ea9f81: Deleting consensus metadata
I20260928 06:28:42.624504 28041 catalog_manager.cc:4841] TS 7fbd1aa844ca4e5e9a8e569671ea9f81 (127.22.225.131:42289): tablet dc8f6e53bd4e47e9822793d8b8ba6bd5 (table test-table [id=eb5e54b4c513425ba8595bed60780c50]) successfully deleted
W20260928 06:28:42.628332 28042 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.225.129:34599 (error 108)
W20260928 06:28:42.628940 28518 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:28:42.628712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.129:34599 (local address 127.0.0.1:52272)
0928 06:28:42.628770 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:42.628782 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:42.628819 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:42.628885 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.129:34599: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 06:28:42.629086 28042 catalog_manager.cc:4584] TS 282544e067434199ad84e68c3d542d64 (127.22.225.129:34599): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dc8f6e53bd4e47e9822793d8b8ba6bd5: Network error: Client connection negotiation failed: client connection to 127.22.225.129:34599: BlockingWrite error: write error: Connection reset by peer (error 104)
I20260928 06:28:42.629362 28547 tablet_replica.cc:330] T dc8f6e53bd4e47e9822793d8b8ba6bd5 P 5fc5a7fb391a49859244a0df091764df: stopping tablet replica
I20260928 06:28:42.629477 28547 raft_consensus.cc:2229] T dc8f6e53bd4e47e9822793d8b8ba6bd5 P 5fc5a7fb391a49859244a0df091764df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:28:42.629603 28547 raft_consensus.cc:2258] T dc8f6e53bd4e47e9822793d8b8ba6bd5 P 5fc5a7fb391a49859244a0df091764df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:28:42.629798 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 28249
I20260928 06:28:42.640717 28518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:42.640533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.130:42501 (local address 127.0.0.1:37612)
0928 06:28:42.640617 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:42.640664 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.130:42501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260928 06:28:42.640784 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 28380
I20260928 06:28:42.645679 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 28026
2026-09-28T06:28:42Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (675 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 06:28:42.666301 23430 mini_oidc.cc:152] Starting JWKS server
I20260928 06:28:42.667472 23430 webserver.cc:460] Webserver started at https://127.0.0.1:43813/ using document root <none> and password file <none>
I20260928 06:28:42.667567 23430 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 06:28:42.667778 23430 webserver.cc:460] Webserver started at http://127.0.0.1:45723/ using document root <none> and password file <none>
2026-09-28T06:28:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:28:42Z Disabled control of system clock
I20260928 06:28:42.675813 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790576899143097-23430-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790576899143097-23430-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790576899143097-23430-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:37889
--webserver_interface=127.22.225.190
--webserver_port=0
--builtin_ntp_servers=127.22.225.148:36661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.225.190:37889
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43813/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790576899143097-23430-0/testchainca.cert with env {}
W20260928 06:28:42.758193 28554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:28:42.758402 28554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:28:42.758424 28554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:28:42.759613 28554 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:28:42.759653 28554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:28:42.759666 28554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:28:42.759678 28554 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:28:42.759689 28554 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 06:28:42.759724 28554 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 06:28:42.759749 28554 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43813/jwks.json
I20260928 06:28:42.761349 28554 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:36661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790576899143097-23430-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790576899143097-23430-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.225.190:37889
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:37889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43813/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.225.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790576899143097-23430-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790576899143097-23430-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:28:42.761597 28554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:28:42.761823 28554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:28:42.766664 28559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:42.766870 28554 server_base.cc:1034] running on GCE node
W20260928 06:28:42.766656 28560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:42.766659 28562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:42.767206 28554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:28:42.767546 28554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:28:42.768641 28554 hybrid_clock.cc:648] HybridClock initialized: now 1790576922768623 us; error 33 us; skew 500 ppm
I20260928 06:28:42.769943 28554 webserver.cc:460] Webserver started at http://127.22.225.190:33191/ using document root <none> and password file <none>
I20260928 06:28:42.770216 28554 fs_manager.cc:359] Metadata directory not provided
I20260928 06:28:42.770288 28554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:28:42.770413 28554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:28:42.771401 28554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790576899143097-23430-0/minicluster-data/master-0/data/instance:
uuid: "d812aa8ad71e4b7487acad8931c3a376"
format_stamp: "Formatted at 2026-09-28 06:28:42 on dist-test-slave-d59m"
server_key: "7d275a3a74192d256990fdd68e1e1c9e"
server_key_iv: "6bf5312af29862d9b6bbc73a7a2676fb"
server_key_version: "encryptionkey@0"
I20260928 06:28:42.771752 28554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790576899143097-23430-0/minicluster-data/master-0/wal/instance:
uuid: "d812aa8ad71e4b7487acad8931c3a376"
format_stamp: "Formatted at 2026-09-28 06:28:42 on dist-test-slave-d59m"
server_key: "7d275a3a74192d256990fdd68e1e1c9e"
server_key_iv: "6bf5312af29862d9b6bbc73a7a2676fb"
server_key_version: "encryptionkey@0"
I20260928 06:28:42.773136 28554 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:28:42.773881 28568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:42.774052 28554 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:28:42.774173 28554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790576899143097-23430-0/minicluster-data/master-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790576899143097-23430-0/minicluster-data/master-0/wal
uuid: "d812aa8ad71e4b7487acad8931c3a376"
format_stamp: "Formatted at 2026-09-28 06:28:42 on dist-test-slave-d59m"
server_key: "7d275a3a74192d256990fdd68e1e1c9e"
server_key_iv: "6bf5312af29862d9b6bbc73a7a2676fb"
server_key_version: "encryptionkey@0"
I20260928 06:28:42.774245 28554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790576899143097-23430-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790576899143097-23430-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790576899143097-23430-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:28:42.784704 28554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:28:42.785012 28554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:28:42.785162 28554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:28:42.789575 28554 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.190:37889
I20260928 06:28:42.789606 28620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.190:37889 every 8 connection(s)
I20260928 06:28:42.789940 28554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
I20260928 06:28:42.790685 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 28554
I20260928 06:28:42.790796 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790576899143097-23430-0/minicluster-data/master-0/wal/instance
I20260928 06:28:42.791011 23430 external_mini_cluster.cc:1428] Setting key 570d70105e33070f43bad7fca43436b4
I20260928 06:28:42.791069 28621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:42.794217 28621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d812aa8ad71e4b7487acad8931c3a376: Bootstrap starting.
I20260928 06:28:42.795037 28621 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d812aa8ad71e4b7487acad8931c3a376: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:42.795336 28621 log.cc:824] T 00000000000000000000000000000000 P d812aa8ad71e4b7487acad8931c3a376: Log is configured to *not* fsync() on all Append() calls
I20260928 06:28:42.796105 28621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d812aa8ad71e4b7487acad8931c3a376: No bootstrap required, opened a new log
I20260928 06:28:42.797953 28624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:42.791665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39126 (local address 127.22.225.190:37889)
0928 06:28:42.791876 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:42.791885 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:42.791918 (+ 33us) server_negotiation.cc:407] Connection header received
0928 06:28:42.792661 (+ 743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:42.792672 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:42.792754 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:42.792899 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:42.793161 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:42.794450 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:42.795618 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:42.796150 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:42.796909 (+ 759us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:42.797200 (+ 291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:42.797203 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:42.797204 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:28:42.797404 (+ 200us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:42.797441 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:28:42.797455 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:42.797536 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:42.797657 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 06:28:42.797746 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":80,"threads_started":1}
I20260928 06:28:42.798954 23430 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:28:42.798909 28621 raft_consensus.cc:348] T 00000000000000000000000000000000 P d812aa8ad71e4b7487acad8931c3a376 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d812aa8ad71e4b7487acad8931c3a376" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 37889 } }
I20260928 06:28:42.799059 28621 raft_consensus.cc:374] T 00000000000000000000000000000000 P d812aa8ad71e4b7487acad8931c3a376 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:42.799083 28621 raft_consensus.cc:729] T 00000000000000000000000000000000 P d812aa8ad71e4b7487acad8931c3a376 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d812aa8ad71e4b7487acad8931c3a376, State: Initialized, Role: FOLLOWER
I20260928 06:28:42.799193 28621 consensus_queue.cc:260] T 00000000000000000000000000000000 P d812aa8ad71e4b7487acad8931c3a376 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d812aa8ad71e4b7487acad8931c3a376" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 37889 } }
I20260928 06:28:42.799258 28621 raft_consensus.cc:388] T 00000000000000000000000000000000 P d812aa8ad71e4b7487acad8931c3a376 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:28:42.799279 28621 raft_consensus.cc:482] T 00000000000000000000000000000000 P d812aa8ad71e4b7487acad8931c3a376 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:28:42.799306 28621 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d812aa8ad71e4b7487acad8931c3a376 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:42.799799 28621 raft_consensus.cc:504] T 00000000000000000000000000000000 P d812aa8ad71e4b7487acad8931c3a376 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d812aa8ad71e4b7487acad8931c3a376" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 37889 } }
I20260928 06:28:42.799916 28621 leader_election.cc:302] T 00000000000000000000000000000000 P d812aa8ad71e4b7487acad8931c3a376 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d812aa8ad71e4b7487acad8931c3a376; no voters:
I20260928 06:28:42.800112 28621 leader_election.cc:288] T 00000000000000000000000000000000 P d812aa8ad71e4b7487acad8931c3a376 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:28:42.800144 28626 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d812aa8ad71e4b7487acad8931c3a376 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:28:42.800371 28626 raft_consensus.cc:686] T 00000000000000000000000000000000 P d812aa8ad71e4b7487acad8931c3a376 [term 1 LEADER]: Becoming Leader. State: Replica: d812aa8ad71e4b7487acad8931c3a376, State: Running, Role: LEADER
I20260928 06:28:42.800417 28621 sys_catalog.cc:564] T 00000000000000000000000000000000 P d812aa8ad71e4b7487acad8931c3a376 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:28:42.800457 28626 consensus_queue.cc:237] T 00000000000000000000000000000000 P d812aa8ad71e4b7487acad8931c3a376 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d812aa8ad71e4b7487acad8931c3a376" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 37889 } }
I20260928 06:28:42.800798 28626 sys_catalog.cc:455] T 00000000000000000000000000000000 P d812aa8ad71e4b7487acad8931c3a376 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d812aa8ad71e4b7487acad8931c3a376. Latest consensus state: current_term: 1 leader_uuid: "d812aa8ad71e4b7487acad8931c3a376" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d812aa8ad71e4b7487acad8931c3a376" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 37889 } } }
I20260928 06:28:42.800899 28626 sys_catalog.cc:458] T 00000000000000000000000000000000 P d812aa8ad71e4b7487acad8931c3a376 [sys.catalog]: This master's current role is: LEADER
I20260928 06:28:42.800812 28628 sys_catalog.cc:455] T 00000000000000000000000000000000 P d812aa8ad71e4b7487acad8931c3a376 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d812aa8ad71e4b7487acad8931c3a376" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d812aa8ad71e4b7487acad8931c3a376" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 37889 } } }
I20260928 06:28:42.801041 28628 sys_catalog.cc:458] T 00000000000000000000000000000000 P d812aa8ad71e4b7487acad8931c3a376 [sys.catalog]: This master's current role is: LEADER
I20260928 06:28:42.801117 28632 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:28:42.801647 28632 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:28:42.803097 28643 mvcc.cc:205] Tried to move back new op lower bound from 7334203075799248896 to 7334203075791056896. Current Snapshot: MvccSnapshot[applied={T|T < 7334203075799248896}]
I20260928 06:28:42.803473 28632 catalog_manager.cc:1347] Generated new cluster ID: 96ec82f47dc44af38419b4b63e30535f
I20260928 06:28:42.803522 28632 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:28:42.807554 28632 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:28:42.808048 28632 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:28:42.810745 28632 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d812aa8ad71e4b7487acad8931c3a376: Generated new TSK 0
I20260928 06:28:42.810923 28632 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 06:28:42.831674 28651 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 06:28:42.831784 28651 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 06:28:42.832244 28624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:42.816822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39132 (local address 127.22.225.190:37889)
0928 06:28:42.816868 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:42.816871 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:42.817069 (+ 198us) server_negotiation.cc:407] Connection header received
0928 06:28:42.817163 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:42.817164 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:42.817175 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:42.817258 (+ 83us) server_negotiation.cc:226] Negotiated authn=JWT
0928 06:28:42.817425 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:42.817883 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:42.818755 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:42.818948 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:42.819047 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:42.819289 (+ 242us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 06:28:42.831957 (+ 12668us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43813/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 06:28:42.832148 (+ 191us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39132: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43813/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 06:28:42.832541 28624 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39132: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43813/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260928 06:28:42.832602 28650 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:28:42.816792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:37889 (local address 127.0.0.1:39132)
0928 06:28:42.816973 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:42.816987 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:42.817076 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:42.817271 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:42.817272 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:42.817275 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0928 06:28:42.817340 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:42.817342 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:42.818029 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:42.818030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:42.818664 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:42.818666 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:42.819081 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:42.819082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:42.819149 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:42.819166 (+ 17us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 06:28:42.832099 (+ 12933us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43813/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 06:28:42.832207 (+ 108us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.22.225.190:37889: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43813/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":85,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.22.225.190:37889: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43813/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 06:28:42.833361 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 28554
2026-09-28T06:28:42Z chronyd exiting
I20260928 06:28:42.869581 23430 test_util.cc:175] -----------------------------------------------
I20260928 06:28:42.869657 23430 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790576899143097-23430-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (204 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 06:28:42.870419 23430 mini_oidc.cc:152] Starting JWKS server
I20260928 06:28:42.871207 23430 webserver.cc:460] Webserver started at https://127.0.0.1:46513/ using document root <none> and password file <none>
I20260928 06:28:42.871261 23430 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 06:28:42.871443 23430 webserver.cc:460] Webserver started at http://127.0.0.1:41885/ using document root <none> and password file <none>
2026-09-28T06:28:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:28:42Z Disabled control of system clock
I20260928 06:28:42.879240 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790576899143097-23430-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790576899143097-23430-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790576899143097-23430-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:37247
--webserver_interface=127.22.225.190
--webserver_port=0
--builtin_ntp_servers=127.22.225.148:46465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.225.190:37247
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46513/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790576899143097-23430-0/testchainca.cert with env {}
W20260928 06:28:42.959198 28657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:28:42.959444 28657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:28:42.959484 28657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:28:42.960682 28657 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:28:42.960734 28657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:28:42.960754 28657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:28:42.960775 28657 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:28:42.960803 28657 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 06:28:42.960832 28657 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 06:28:42.960856 28657 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46513/jwks.json
I20260928 06:28:42.962299 28657 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:46465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790576899143097-23430-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790576899143097-23430-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.225.190:37247
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:37247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46513/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.225.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790576899143097-23430-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790576899143097-23430-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:28:42.962574 28657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:28:42.962785 28657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:28:42.968154 28662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:42.968154 28665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:42.968370 28657 server_base.cc:1034] running on GCE node
W20260928 06:28:42.968160 28663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:42.968716 28657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:28:42.969051 28657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:28:42.970206 28657 hybrid_clock.cc:648] HybridClock initialized: now 1790576922970183 us; error 42 us; skew 500 ppm
I20260928 06:28:42.976825 28657 webserver.cc:460] Webserver started at http://127.22.225.190:41973/ using document root <none> and password file <none>
I20260928 06:28:42.977089 28657 fs_manager.cc:359] Metadata directory not provided
I20260928 06:28:42.977160 28657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:28:42.977309 28657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:28:42.978368 28657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790576899143097-23430-0/minicluster-data/master-0/data/instance:
uuid: "88c63f1e84d1496795547422afee9b99"
format_stamp: "Formatted at 2026-09-28 06:28:42 on dist-test-slave-d59m"
server_key: "6e3bdae785af5c8964d3479d5efe942a"
server_key_iv: "4728d2841a9791f98aac292d24ac6951"
server_key_version: "encryptionkey@0"
I20260928 06:28:42.978727 28657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790576899143097-23430-0/minicluster-data/master-0/wal/instance:
uuid: "88c63f1e84d1496795547422afee9b99"
format_stamp: "Formatted at 2026-09-28 06:28:42 on dist-test-slave-d59m"
server_key: "6e3bdae785af5c8964d3479d5efe942a"
server_key_iv: "4728d2841a9791f98aac292d24ac6951"
server_key_version: "encryptionkey@0"
I20260928 06:28:42.980089 28657 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:28:42.980856 28671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:42.981036 28657 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:28:42.981148 28657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790576899143097-23430-0/minicluster-data/master-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790576899143097-23430-0/minicluster-data/master-0/wal
uuid: "88c63f1e84d1496795547422afee9b99"
format_stamp: "Formatted at 2026-09-28 06:28:42 on dist-test-slave-d59m"
server_key: "6e3bdae785af5c8964d3479d5efe942a"
server_key_iv: "4728d2841a9791f98aac292d24ac6951"
server_key_version: "encryptionkey@0"
I20260928 06:28:42.981225 28657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790576899143097-23430-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790576899143097-23430-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790576899143097-23430-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:28:42.990684 28657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:28:42.990955 28657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:28:42.991109 28657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:28:42.995630 28657 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.190:37247
I20260928 06:28:42.995652 28723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.190:37247 every 8 connection(s)
I20260928 06:28:42.995975 28657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
I20260928 06:28:42.996694 28724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:42.999400 28724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88c63f1e84d1496795547422afee9b99: Bootstrap starting.
I20260928 06:28:42.999980 28724 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88c63f1e84d1496795547422afee9b99: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:43.000291 28724 log.cc:824] T 00000000000000000000000000000000 P 88c63f1e84d1496795547422afee9b99: Log is configured to *not* fsync() on all Append() calls
I20260928 06:28:43.001140 28724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88c63f1e84d1496795547422afee9b99: No bootstrap required, opened a new log
I20260928 06:28:43.002769 28724 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88c63f1e84d1496795547422afee9b99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88c63f1e84d1496795547422afee9b99" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 37247 } }
I20260928 06:28:43.002909 28724 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88c63f1e84d1496795547422afee9b99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:43.002933 28724 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88c63f1e84d1496795547422afee9b99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88c63f1e84d1496795547422afee9b99, State: Initialized, Role: FOLLOWER
I20260928 06:28:43.003082 28724 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88c63f1e84d1496795547422afee9b99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88c63f1e84d1496795547422afee9b99" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 37247 } }
I20260928 06:28:43.003144 28724 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88c63f1e84d1496795547422afee9b99 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:28:43.003192 28724 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88c63f1e84d1496795547422afee9b99 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:28:43.003240 28724 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88c63f1e84d1496795547422afee9b99 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:43.003364 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 28657
I20260928 06:28:43.003470 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790576899143097-23430-0/minicluster-data/master-0/wal/instance
I20260928 06:28:43.003639 23430 external_mini_cluster.cc:1428] Setting key 4411f0cdaf8576a34ef96db774d4be00
I20260928 06:28:43.003803 28724 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88c63f1e84d1496795547422afee9b99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88c63f1e84d1496795547422afee9b99" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 37247 } }
I20260928 06:28:43.003912 28724 leader_election.cc:302] T 00000000000000000000000000000000 P 88c63f1e84d1496795547422afee9b99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88c63f1e84d1496795547422afee9b99; no voters:
I20260928 06:28:43.004093 28724 leader_election.cc:288] T 00000000000000000000000000000000 P 88c63f1e84d1496795547422afee9b99 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:28:43.004191 28727 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88c63f1e84d1496795547422afee9b99 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:28:43.004367 28724 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88c63f1e84d1496795547422afee9b99 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:28:43.004390 28727 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88c63f1e84d1496795547422afee9b99 [term 1 LEADER]: Becoming Leader. State: Replica: 88c63f1e84d1496795547422afee9b99, State: Running, Role: LEADER
I20260928 06:28:43.004627 28727 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88c63f1e84d1496795547422afee9b99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88c63f1e84d1496795547422afee9b99" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 37247 } }
I20260928 06:28:43.005035 28727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88c63f1e84d1496795547422afee9b99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88c63f1e84d1496795547422afee9b99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88c63f1e84d1496795547422afee9b99" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 37247 } } }
I20260928 06:28:43.005165 28727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88c63f1e84d1496795547422afee9b99 [sys.catalog]: This master's current role is: LEADER
I20260928 06:28:43.005327 28730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88c63f1e84d1496795547422afee9b99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88c63f1e84d1496795547422afee9b99. Latest consensus state: current_term: 1 leader_uuid: "88c63f1e84d1496795547422afee9b99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88c63f1e84d1496795547422afee9b99" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 37247 } } }
I20260928 06:28:43.005420 28730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88c63f1e84d1496795547422afee9b99 [sys.catalog]: This master's current role is: LEADER
W20260928 06:28:43.006254 28744 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 88c63f1e84d1496795547422afee9b99: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:28:43.006352 28744 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:28:43.006455 28738 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:28:43.006640 28738 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:28:43.008196 28738 catalog_manager.cc:1347] Generated new cluster ID: 3698a600939246c1a1425a85fda84071
I20260928 06:28:43.008242 28738 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:28:43.011097 28731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:43.004335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39046 (local address 127.22.225.190:37247)
0928 06:28:43.004542 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:43.004555 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:43.004621 (+ 66us) server_negotiation.cc:407] Connection header received
0928 06:28:43.004742 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:43.004750 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:43.004828 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:43.005014 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:43.005523 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:43.007199 (+ 1676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:43.008327 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:43.008947 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:43.009394 (+ 447us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:43.010191 (+ 797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:43.010194 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:43.010195 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:28:43.010399 (+ 204us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:43.010443 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:28:43.010458 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:43.010763 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:43.010795 (+ 32us) server_negotiation.cc:299] Negotiation successful
0928 06:28:43.010891 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":78,"threads_started":1}
I20260928 06:28:43.012116 23430 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:28:43.042469 28738 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:28:43.043169 28738 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:28:43.047015 28738 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88c63f1e84d1496795547422afee9b99: Generated new TSK 0
I20260928 06:28:43.047184 28738 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 06:28:43.084134 28754 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 06:28:43.084296 28754 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260928 06:28:43.084734 28753 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:28:43.061408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:37247 (local address 127.0.0.1:39052)
0928 06:28:43.061562 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:43.061573 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:43.061657 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:43.061825 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:43.061826 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:43.061828 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 06:28:43.061891 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:43.061893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:43.062815 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:43.062816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:43.063777 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:43.063779 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:43.064280 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:43.064281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:43.064344 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:43.064349 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 06:28:43.084467 (+ 20118us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46513/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 06:28:43.084613 (+ 146us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.22.225.190:37247: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46513/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":98,"threads_started":1}
I20260928 06:28:43.085091 28731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:43.061483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39052 (local address 127.22.225.190:37247)
0928 06:28:43.061541 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:43.061543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:43.061650 (+ 107us) server_negotiation.cc:407] Connection header received
0928 06:28:43.061714 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:43.061716 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:43.061729 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:43.061811 (+ 82us) server_negotiation.cc:226] Negotiated authn=JWT
0928 06:28:43.061965 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:43.062704 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:43.063875 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:43.064194 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:43.064267 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:43.064456 (+ 189us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 06:28:43.084315 (+ 19859us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46513/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 06:28:43.085027 (+ 712us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39052: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46513/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":33}
W20260928 06:28:43.085147 28731 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39052: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46513/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260928 06:28:43.085559 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 28657
2026-09-28T06:28:43Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (253 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 06:28:43.123700 23430 mini_oidc.cc:152] Starting JWKS server
I20260928 06:28:43.124483 23430 webserver.cc:460] Webserver started at https://127.0.0.1:42883/ using document root <none> and password file <none>
I20260928 06:28:43.124548 23430 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 06:28:43.124739 23430 webserver.cc:460] Webserver started at http://127.0.0.1:39049/ using document root <none> and password file <none>
2026-09-28T06:28:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:28:43Z Disabled control of system clock
I20260928 06:28:43.132652 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790576899143097-23430-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790576899143097-23430-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790576899143097-23430-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:34037
--webserver_interface=127.22.225.190
--webserver_port=0
--builtin_ntp_servers=127.22.225.148:44697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.225.190:34037
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42883/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 06:28:43.213126 28760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:28:43.213352 28760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:28:43.213389 28760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:28:43.214567 28760 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:28:43.214632 28760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:28:43.214656 28760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:28:43.214687 28760 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:28:43.214735 28760 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 06:28:43.214766 28760 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 06:28:43.214792 28760 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42883/jwks.json
I20260928 06:28:43.216135 28760 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:44697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790576899143097-23430-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790576899143097-23430-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.225.190:34037
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:34037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42883/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28760
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790576899143097-23430-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:28:43.216454 28760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:28:43.216786 28760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:28:43.221645 28766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:43.221645 28768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:43.221870 28760 server_base.cc:1034] running on GCE node
W20260928 06:28:43.221874 28765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:43.222345 28760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:28:43.222656 28760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:28:43.223820 28760 hybrid_clock.cc:648] HybridClock initialized: now 1790576923223808 us; error 33 us; skew 500 ppm
I20260928 06:28:43.225174 28760 webserver.cc:460] Webserver started at http://127.22.225.190:34611/ using document root <none> and password file <none>
I20260928 06:28:43.225441 28760 fs_manager.cc:359] Metadata directory not provided
I20260928 06:28:43.225512 28760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:28:43.225665 28760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:28:43.226716 28760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790576899143097-23430-0/minicluster-data/master-0/data/instance:
uuid: "ea754c32006b4e7d80fab5163bc86c37"
format_stamp: "Formatted at 2026-09-28 06:28:43 on dist-test-slave-d59m"
server_key: "2b993613e019fdcab299abf6874f4139"
server_key_iv: "a014be4ed93ac318152a5c5f03a62df2"
server_key_version: "encryptionkey@0"
I20260928 06:28:43.227083 28760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790576899143097-23430-0/minicluster-data/master-0/wal/instance:
uuid: "ea754c32006b4e7d80fab5163bc86c37"
format_stamp: "Formatted at 2026-09-28 06:28:43 on dist-test-slave-d59m"
server_key: "2b993613e019fdcab299abf6874f4139"
server_key_iv: "a014be4ed93ac318152a5c5f03a62df2"
server_key_version: "encryptionkey@0"
I20260928 06:28:43.228448 28760 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:28:43.229279 28774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:43.229473 28760 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:28:43.229573 28760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790576899143097-23430-0/minicluster-data/master-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790576899143097-23430-0/minicluster-data/master-0/wal
uuid: "ea754c32006b4e7d80fab5163bc86c37"
format_stamp: "Formatted at 2026-09-28 06:28:43 on dist-test-slave-d59m"
server_key: "2b993613e019fdcab299abf6874f4139"
server_key_iv: "a014be4ed93ac318152a5c5f03a62df2"
server_key_version: "encryptionkey@0"
I20260928 06:28:43.229656 28760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790576899143097-23430-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790576899143097-23430-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790576899143097-23430-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:28:43.250715 28760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:28:43.251071 28760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:28:43.251224 28760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:28:43.255419 28760 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.190:34037
I20260928 06:28:43.255473 28826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.190:34037 every 8 connection(s)
I20260928 06:28:43.255813 28760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
I20260928 06:28:43.256604 28827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:43.256970 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 28760
I20260928 06:28:43.257073 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790576899143097-23430-0/minicluster-data/master-0/wal/instance
I20260928 06:28:43.257230 23430 external_mini_cluster.cc:1428] Setting key 01b31c39ca33d7e098b381dcad656b13
I20260928 06:28:43.260385 28827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea754c32006b4e7d80fab5163bc86c37: Bootstrap starting.
I20260928 06:28:43.261178 28827 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea754c32006b4e7d80fab5163bc86c37: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:43.261482 28827 log.cc:824] T 00000000000000000000000000000000 P ea754c32006b4e7d80fab5163bc86c37: Log is configured to *not* fsync() on all Append() calls
I20260928 06:28:43.262276 28827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea754c32006b4e7d80fab5163bc86c37: No bootstrap required, opened a new log
I20260928 06:28:43.263957 28830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:43.257862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57854 (local address 127.22.225.190:34037)
0928 06:28:43.258091 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:43.258100 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:43.258551 (+ 451us) server_negotiation.cc:407] Connection header received
0928 06:28:43.259149 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:43.259158 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:43.259248 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:43.259400 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:43.259728 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:43.260973 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:43.261938 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:43.262360 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:43.262456 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:43.263266 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:43.263270 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:43.263270 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:28:43.263438 (+ 168us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:43.263470 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:28:43.263480 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:43.263556 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:43.263652 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 06:28:43.263737 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":97,"threads_started":1}
I20260928 06:28:43.264853 23430 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:28:43.264962 28827 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea754c32006b4e7d80fab5163bc86c37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea754c32006b4e7d80fab5163bc86c37" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 34037 } }
I20260928 06:28:43.265159 28827 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea754c32006b4e7d80fab5163bc86c37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:43.265203 28827 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea754c32006b4e7d80fab5163bc86c37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea754c32006b4e7d80fab5163bc86c37, State: Initialized, Role: FOLLOWER
I20260928 06:28:43.265326 28827 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea754c32006b4e7d80fab5163bc86c37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea754c32006b4e7d80fab5163bc86c37" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 34037 } }
I20260928 06:28:43.265403 28827 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea754c32006b4e7d80fab5163bc86c37 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:28:43.265432 28827 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea754c32006b4e7d80fab5163bc86c37 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:28:43.265478 28827 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea754c32006b4e7d80fab5163bc86c37 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:43.266194 28827 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea754c32006b4e7d80fab5163bc86c37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea754c32006b4e7d80fab5163bc86c37" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 34037 } }
I20260928 06:28:43.266299 28827 leader_election.cc:302] T 00000000000000000000000000000000 P ea754c32006b4e7d80fab5163bc86c37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea754c32006b4e7d80fab5163bc86c37; no voters:
I20260928 06:28:43.266456 28827 leader_election.cc:288] T 00000000000000000000000000000000 P ea754c32006b4e7d80fab5163bc86c37 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:28:43.266512 28832 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea754c32006b4e7d80fab5163bc86c37 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:28:43.266670 28832 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea754c32006b4e7d80fab5163bc86c37 [term 1 LEADER]: Becoming Leader. State: Replica: ea754c32006b4e7d80fab5163bc86c37, State: Running, Role: LEADER
I20260928 06:28:43.266703 28827 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea754c32006b4e7d80fab5163bc86c37 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:28:43.266765 28832 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea754c32006b4e7d80fab5163bc86c37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea754c32006b4e7d80fab5163bc86c37" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 34037 } }
I20260928 06:28:43.267125 28833 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea754c32006b4e7d80fab5163bc86c37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea754c32006b4e7d80fab5163bc86c37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea754c32006b4e7d80fab5163bc86c37" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 34037 } } }
I20260928 06:28:43.267207 28833 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea754c32006b4e7d80fab5163bc86c37 [sys.catalog]: This master's current role is: LEADER
I20260928 06:28:43.267591 28834 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea754c32006b4e7d80fab5163bc86c37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea754c32006b4e7d80fab5163bc86c37. Latest consensus state: current_term: 1 leader_uuid: "ea754c32006b4e7d80fab5163bc86c37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea754c32006b4e7d80fab5163bc86c37" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 34037 } } }
I20260928 06:28:43.267858 28834 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea754c32006b4e7d80fab5163bc86c37 [sys.catalog]: This master's current role is: LEADER
W20260928 06:28:43.268544 28848 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ea754c32006b4e7d80fab5163bc86c37: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:28:43.268615 28848 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:28:43.268692 28839 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:28:43.268865 28839 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:28:43.270421 28839 catalog_manager.cc:1347] Generated new cluster ID: f0c3a32a057048799a0a176b517fc123
I20260928 06:28:43.270471 28839 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:28:43.301508 28839 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:28:43.302186 28839 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:28:43.304306 28839 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea754c32006b4e7d80fab5163bc86c37: Generated new TSK 0
I20260928 06:28:43.304471 28839 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:28:43.315387 28830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:43.311264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57866 (local address 127.22.225.190:34037)
0928 06:28:43.311303 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:43.311305 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:43.311431 (+ 126us) server_negotiation.cc:407] Connection header received
0928 06:28:43.311487 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:43.311488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:43.311500 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:43.311583 (+ 83us) server_negotiation.cc:226] Negotiated authn=JWT
0928 06:28:43.311725 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:43.312528 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:43.313610 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:43.313926 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:43.314007 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:43.314192 (+ 185us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 06:28:43.315169 (+ 977us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42883/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 06:28:43.315312 (+ 143us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57866: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42883/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 06:28:43.315448 28830 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57866: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42883/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260928 06:28:43.315464 28856 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:28:43.311204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:34037 (local address 127.0.0.1:57866)
0928 06:28:43.311353 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:43.311364 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:43.311430 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:43.311593 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:43.311593 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:43.311596 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0928 06:28:43.311654 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:43.311656 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:43.312666 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:43.312667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:43.313516 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:43.313519 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:43.314027 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:43.314028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:43.314080 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:43.314087 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 06:28:43.315269 (+ 1182us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42883/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 06:28:43.315373 (+ 104us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.22.225.190:34037: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42883/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":100,"threads_started":1}
I20260928 06:28:43.316094 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 28760
E20260928 06:28:43.328296 28857 webserver.cc:533] Webserver: sslize failed:
E20260928 06:28:43.328372 28857 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-28T06:28:43Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (231 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T06:28:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:28:43Z Disabled control of system clock
I20260928 06:28:43.363055 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790576899143097-23430-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790576899143097-23430-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790576899143097-23430-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:34685
--webserver_interface=127.22.225.190
--webserver_port=0
--builtin_ntp_servers=127.22.225.148:44047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.225.190:34685
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790576899143097-23430-0/testchainca.cert with env {}
W20260928 06:28:43.443553 28861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:28:43.443746 28861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:28:43.443768 28861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:28:43.444937 28861 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:28:43.444976 28861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:28:43.444988 28861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:28:43.445000 28861 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:28:43.445048 28861 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 06:28:43.445065 28861 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 06:28:43.445091 28861 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 06:28:43.446553 28861 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:44047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790576899143097-23430-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790576899143097-23430-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.225.190:34685
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:34685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.225.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790576899143097-23430-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28861
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790576899143097-23430-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:28:43.446792 28861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:28:43.447006 28861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:28:43.451732 28867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:43.451753 28866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:43.451898 28869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:43.452216 28861 server_base.cc:1034] running on GCE node
I20260928 06:28:43.452468 28861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:28:43.452800 28861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:28:43.453962 28861 hybrid_clock.cc:648] HybridClock initialized: now 1790576923453939 us; error 41 us; skew 500 ppm
I20260928 06:28:43.455150 28861 webserver.cc:460] Webserver started at http://127.22.225.190:37941/ using document root <none> and password file <none>
I20260928 06:28:43.455382 28861 fs_manager.cc:359] Metadata directory not provided
I20260928 06:28:43.455461 28861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:28:43.455590 28861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:28:43.456699 28861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790576899143097-23430-0/minicluster-data/master-0/data/instance:
uuid: "c7498072234c4f2cb076d7dfdd478a6d"
format_stamp: "Formatted at 2026-09-28 06:28:43 on dist-test-slave-d59m"
server_key: "eb4827fc5ce8bb8d214dc231f29ef704"
server_key_iv: "2b1ccd6cbf0da47eecf845d2ac17bab4"
server_key_version: "encryptionkey@0"
I20260928 06:28:43.457054 28861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790576899143097-23430-0/minicluster-data/master-0/wal/instance:
uuid: "c7498072234c4f2cb076d7dfdd478a6d"
format_stamp: "Formatted at 2026-09-28 06:28:43 on dist-test-slave-d59m"
server_key: "eb4827fc5ce8bb8d214dc231f29ef704"
server_key_iv: "2b1ccd6cbf0da47eecf845d2ac17bab4"
server_key_version: "encryptionkey@0"
I20260928 06:28:43.458407 28861 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 06:28:43.459152 28875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:43.459324 28861 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:28:43.459450 28861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790576899143097-23430-0/minicluster-data/master-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790576899143097-23430-0/minicluster-data/master-0/wal
uuid: "c7498072234c4f2cb076d7dfdd478a6d"
format_stamp: "Formatted at 2026-09-28 06:28:43 on dist-test-slave-d59m"
server_key: "eb4827fc5ce8bb8d214dc231f29ef704"
server_key_iv: "2b1ccd6cbf0da47eecf845d2ac17bab4"
server_key_version: "encryptionkey@0"
I20260928 06:28:43.459528 28861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790576899143097-23430-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790576899143097-23430-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790576899143097-23430-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:28:43.471298 28861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:28:43.471612 28861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:28:43.471758 28861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:28:43.476423 28861 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.190:34685
I20260928 06:28:43.476452 28927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.190:34685 every 8 connection(s)
I20260928 06:28:43.476861 28861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
I20260928 06:28:43.477152 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 28861
I20260928 06:28:43.477252 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790576899143097-23430-0/minicluster-data/master-0/wal/instance
I20260928 06:28:43.477438 23430 external_mini_cluster.cc:1428] Setting key c1620dd676c291a70b67e81bd8b4dd2e
I20260928 06:28:43.477969 28928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:43.481509 28928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7498072234c4f2cb076d7dfdd478a6d: Bootstrap starting.
I20260928 06:28:43.482180 28928 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7498072234c4f2cb076d7dfdd478a6d: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:43.482504 28928 log.cc:824] T 00000000000000000000000000000000 P c7498072234c4f2cb076d7dfdd478a6d: Log is configured to *not* fsync() on all Append() calls
I20260928 06:28:43.483379 28928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7498072234c4f2cb076d7dfdd478a6d: No bootstrap required, opened a new log
I20260928 06:28:43.484539 28931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:43.478186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44008 (local address 127.22.225.190:34685)
0928 06:28:43.478449 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:43.478458 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:43.478520 (+ 62us) server_negotiation.cc:407] Connection header received
0928 06:28:43.479126 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:43.479135 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:43.479222 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:43.479384 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:43.479641 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:43.481028 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:43.482030 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:43.482712 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:43.482835 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:43.483590 (+ 755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:43.483594 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:43.483594 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:28:43.483825 (+ 231us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:43.483867 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:28:43.483881 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:43.483938 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:43.484206 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 06:28:43.484289 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":103,"threads_started":1}
I20260928 06:28:43.485733 23430 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:28:43.486060 28928 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7498072234c4f2cb076d7dfdd478a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7498072234c4f2cb076d7dfdd478a6d" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 34685 } }
I20260928 06:28:43.486217 28928 raft_consensus.cc:374] T 00000000000000000000000000000000 P c7498072234c4f2cb076d7dfdd478a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:43.486260 28928 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7498072234c4f2cb076d7dfdd478a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7498072234c4f2cb076d7dfdd478a6d, State: Initialized, Role: FOLLOWER
I20260928 06:28:43.486382 28928 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7498072234c4f2cb076d7dfdd478a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7498072234c4f2cb076d7dfdd478a6d" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 34685 } }
I20260928 06:28:43.486471 28928 raft_consensus.cc:388] T 00000000000000000000000000000000 P c7498072234c4f2cb076d7dfdd478a6d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:28:43.486514 28928 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7498072234c4f2cb076d7dfdd478a6d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:28:43.486555 28928 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7498072234c4f2cb076d7dfdd478a6d [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:43.487218 28928 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7498072234c4f2cb076d7dfdd478a6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7498072234c4f2cb076d7dfdd478a6d" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 34685 } }
I20260928 06:28:43.487305 28928 leader_election.cc:302] T 00000000000000000000000000000000 P c7498072234c4f2cb076d7dfdd478a6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c7498072234c4f2cb076d7dfdd478a6d; no voters:
I20260928 06:28:43.487519 28928 leader_election.cc:288] T 00000000000000000000000000000000 P c7498072234c4f2cb076d7dfdd478a6d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:28:43.487571 28933 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7498072234c4f2cb076d7dfdd478a6d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:28:43.487722 28933 raft_consensus.cc:686] T 00000000000000000000000000000000 P c7498072234c4f2cb076d7dfdd478a6d [term 1 LEADER]: Becoming Leader. State: Replica: c7498072234c4f2cb076d7dfdd478a6d, State: Running, Role: LEADER
I20260928 06:28:43.487785 28928 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7498072234c4f2cb076d7dfdd478a6d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:28:43.487814 28933 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7498072234c4f2cb076d7dfdd478a6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7498072234c4f2cb076d7dfdd478a6d" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 34685 } }
I20260928 06:28:43.488193 28934 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7498072234c4f2cb076d7dfdd478a6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c7498072234c4f2cb076d7dfdd478a6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7498072234c4f2cb076d7dfdd478a6d" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 34685 } } }
I20260928 06:28:43.488279 28934 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7498072234c4f2cb076d7dfdd478a6d [sys.catalog]: This master's current role is: LEADER
I20260928 06:28:43.488462 28935 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7498072234c4f2cb076d7dfdd478a6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7498072234c4f2cb076d7dfdd478a6d. Latest consensus state: current_term: 1 leader_uuid: "c7498072234c4f2cb076d7dfdd478a6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7498072234c4f2cb076d7dfdd478a6d" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 34685 } } }
I20260928 06:28:43.488551 28935 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7498072234c4f2cb076d7dfdd478a6d [sys.catalog]: This master's current role is: LEADER
I20260928 06:28:43.488503 28940 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:28:43.489076 28940 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:28:43.490911 28940 catalog_manager.cc:1347] Generated new cluster ID: 6a52f4bfcf554beabe71f63158544127
I20260928 06:28:43.490967 28940 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:28:43.507011 28940 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:28:43.507580 28940 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:28:43.511934 28940 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c7498072234c4f2cb076d7dfdd478a6d: Generated new TSK 0
I20260928 06:28:43.512140 28940 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:28:43.547060 28931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:43.517450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44020 (local address 127.22.225.190:34685)
0928 06:28:43.517501 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:43.517503 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:43.517606 (+ 103us) server_negotiation.cc:407] Connection header received
0928 06:28:43.517681 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:43.517682 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:43.517693 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:43.517762 (+ 69us) server_negotiation.cc:226] Negotiated authn=JWT
0928 06:28:43.517891 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:43.518395 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:43.519207 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:43.519534 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:43.519616 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:43.519801 (+ 185us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 06:28:43.546766 (+ 26965us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 06:28:43.546972 (+ 206us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44020: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":24}
W20260928 06:28:43.547132 28957 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:28:43.517393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:34685 (local address 127.0.0.1:44020)
0928 06:28:43.517540 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:43.517548 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:43.517620 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:43.517761 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:43.517762 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:43.517764 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 06:28:43.517817 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:43.517819 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:43.518513 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:43.518514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:43.519108 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:43.519110 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:43.519625 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:43.519626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:43.519687 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:43.519696 (+ 9us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 06:28:43.546908 (+ 27212us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 06:28:43.547029 (+ 121us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.22.225.190:34685: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":77,"threads_started":1}
W20260928 06:28:43.547145 28931 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44020: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
I20260928 06:28:43.547497 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 28861
2026-09-28T06:28:43Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (209 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (117 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (93 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:28:43 dist-test-slave-d59m krb5kdc[28969](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:28:43 dist-test-slave-d59m krb5kdc[28969](info): set up 2 sockets
Sep 28 06:28:43 dist-test-slave-d59m krb5kdc[28969](info): commencing operation
krb5kdc: starting...
W20260928 06:28:44.829411 23430 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Sep 28 06:28:44 dist-test-slave-d59m krb5kdc[28969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:28:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:28:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab.
Entry for principal kudu/127.22.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab.
I20260928 06:28:44.954985 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:38193
--webserver_interface=127.22.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab
--principal=kudu/127.22.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:42431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.225.190:38193
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:28:45.036690 28982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:28:45.036908 28982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:28:45.036945 28982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:28:45.038162 28982 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:28:45.038235 28982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:28:45.038262 28982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:28:45.038292 28982 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:28:45.038317 28982 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:28:45.039840 28982 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:42431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.225.190:38193
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab
--principal=kudu/127.22.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:38193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:28:45.040103 28982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:28:45.040329 28982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:28:45.045254 28988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:45.045392 28987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:45.045259 28990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:45.045511 28982 server_base.cc:1034] running on GCE node
I20260928 06:28:45.045734 28982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:28:45.046090 28982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:28:45.047272 28982 hybrid_clock.cc:648] HybridClock initialized: now 1790576925047252 us; error 39 us; skew 500 ppm
Sep 28 06:28:45 dist-test-slave-d59m krb5kdc[28969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576925, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:28:45.050028 28982 init.cc:377] Logged in from keytab as kudu/127.22.225.190@KRBTEST.COM (short username kudu)
I20260928 06:28:45.050534 28982 webserver.cc:460] Webserver started at http://127.22.225.190:40271/ using document root <none> and password file <none>
I20260928 06:28:45.050791 28982 fs_manager.cc:359] Metadata directory not provided
I20260928 06:28:45.050848 28982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:28:45.050976 28982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:28:45.052316 28982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/minicluster-data/master-0/data/instance:
uuid: "8fdeded06cb14ebebdaa24dce4e9bafb"
format_stamp: "Formatted at 2026-09-28 06:28:45 on dist-test-slave-d59m"
server_key: "90c04fa2ec2d5294c11ece9497a68866"
server_key_iv: "4b9b7daa8f34205b1e8102643f7dce43"
server_key_version: "encryptionkey@0"
I20260928 06:28:45.052737 28982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/minicluster-data/master-0/wal/instance:
uuid: "8fdeded06cb14ebebdaa24dce4e9bafb"
format_stamp: "Formatted at 2026-09-28 06:28:45 on dist-test-slave-d59m"
server_key: "90c04fa2ec2d5294c11ece9497a68866"
server_key_iv: "4b9b7daa8f34205b1e8102643f7dce43"
server_key_version: "encryptionkey@0"
I20260928 06:28:45.054284 28982 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:28:45.055089 28997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:45.055286 28982 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:28:45.055375 28982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/minicluster-data/master-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/minicluster-data/master-0/wal
uuid: "8fdeded06cb14ebebdaa24dce4e9bafb"
format_stamp: "Formatted at 2026-09-28 06:28:45 on dist-test-slave-d59m"
server_key: "90c04fa2ec2d5294c11ece9497a68866"
server_key_iv: "4b9b7daa8f34205b1e8102643f7dce43"
server_key_version: "encryptionkey@0"
I20260928 06:28:45.055439 28982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:28:45.065948 28982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:28:45.068342 28982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:28:45.068490 28982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:28:45.072706 28982 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.190:38193
I20260928 06:28:45.072770 29049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.190:38193 every 8 connection(s)
I20260928 06:28:45.073155 28982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
I20260928 06:28:45.074015 29050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:45.076303 29050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fdeded06cb14ebebdaa24dce4e9bafb: Bootstrap starting.
I20260928 06:28:45.077080 29050 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8fdeded06cb14ebebdaa24dce4e9bafb: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:45.077380 29050 log.cc:824] T 00000000000000000000000000000000 P 8fdeded06cb14ebebdaa24dce4e9bafb: Log is configured to *not* fsync() on all Append() calls
I20260928 06:28:45.078145 29050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fdeded06cb14ebebdaa24dce4e9bafb: No bootstrap required, opened a new log
I20260928 06:28:45.079689 29050 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8fdeded06cb14ebebdaa24dce4e9bafb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fdeded06cb14ebebdaa24dce4e9bafb" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 38193 } }
I20260928 06:28:45.079843 29050 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8fdeded06cb14ebebdaa24dce4e9bafb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:45.080053 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 28982
I20260928 06:28:45.080160 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/minicluster-data/master-0/wal/instance
I20260928 06:28:45.080338 23430 external_mini_cluster.cc:1428] Setting key baea6588c60778beeb34e4bebd8ca24c
I20260928 06:28:45.080454 29050 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8fdeded06cb14ebebdaa24dce4e9bafb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fdeded06cb14ebebdaa24dce4e9bafb, State: Initialized, Role: FOLLOWER
I20260928 06:28:45.080679 29050 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8fdeded06cb14ebebdaa24dce4e9bafb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fdeded06cb14ebebdaa24dce4e9bafb" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 38193 } }
I20260928 06:28:45.080763 29050 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8fdeded06cb14ebebdaa24dce4e9bafb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:28:45.080792 29050 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8fdeded06cb14ebebdaa24dce4e9bafb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:28:45.080833 29050 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8fdeded06cb14ebebdaa24dce4e9bafb [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:45.081571 29050 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8fdeded06cb14ebebdaa24dce4e9bafb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fdeded06cb14ebebdaa24dce4e9bafb" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 38193 } }
I20260928 06:28:45.081699 29050 leader_election.cc:302] T 00000000000000000000000000000000 P 8fdeded06cb14ebebdaa24dce4e9bafb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8fdeded06cb14ebebdaa24dce4e9bafb; no voters:
I20260928 06:28:45.081873 29050 leader_election.cc:288] T 00000000000000000000000000000000 P 8fdeded06cb14ebebdaa24dce4e9bafb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:28:45.081939 29054 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8fdeded06cb14ebebdaa24dce4e9bafb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:28:45.082129 29050 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8fdeded06cb14ebebdaa24dce4e9bafb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:28:45.082329 29054 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8fdeded06cb14ebebdaa24dce4e9bafb [term 1 LEADER]: Becoming Leader. State: Replica: 8fdeded06cb14ebebdaa24dce4e9bafb, State: Running, Role: LEADER
I20260928 06:28:45.082424 29054 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8fdeded06cb14ebebdaa24dce4e9bafb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fdeded06cb14ebebdaa24dce4e9bafb" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 38193 } }
I20260928 06:28:45.082791 29055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fdeded06cb14ebebdaa24dce4e9bafb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8fdeded06cb14ebebdaa24dce4e9bafb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fdeded06cb14ebebdaa24dce4e9bafb" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 38193 } } }
I20260928 06:28:45.082872 29055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fdeded06cb14ebebdaa24dce4e9bafb [sys.catalog]: This master's current role is: LEADER
I20260928 06:28:45.083048 29056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fdeded06cb14ebebdaa24dce4e9bafb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8fdeded06cb14ebebdaa24dce4e9bafb. Latest consensus state: current_term: 1 leader_uuid: "8fdeded06cb14ebebdaa24dce4e9bafb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fdeded06cb14ebebdaa24dce4e9bafb" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 38193 } } }
I20260928 06:28:45.083118 29056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fdeded06cb14ebebdaa24dce4e9bafb [sys.catalog]: This master's current role is: LEADER
I20260928 06:28:45.083072 29063 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:28:45.083679 29063 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:28:45.087920 29063 catalog_manager.cc:1347] Generated new cluster ID: d79bd75720974286aa2b7872946ada52
I20260928 06:28:45.087986 29063 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:28:45 dist-test-slave-d59m krb5kdc[28969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:28:45.094992 29063 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:28:45.095676 29063 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:28:45.095958 29070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:45.083550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58588 (local address 127.22.225.190:38193)
0928 06:28:45.083701 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:45.083711 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:45.083746 (+ 35us) server_negotiation.cc:407] Connection header received
0928 06:28:45.083869 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:45.083876 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:45.083936 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:45.084113 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:45.085027 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:45.086630 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:45.087721 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:45.088373 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:45.088475 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:45.091532 (+ 3057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:45.091537 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:45.091537 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:45.091550 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:45.093566 (+ 2016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:45.093995 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:45.093998 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:45.093998 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:45.094101 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:45.094312 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:45.094313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:45.094313 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:45.094713 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:45.094943 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:45.095484 (+ 541us) server_negotiation.cc:299] Negotiation successful
0928 06:28:45.095687 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":95,"thread_start_us":64,"threads_started":1}
I20260928 06:28:45.101424 29063 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8fdeded06cb14ebebdaa24dce4e9bafb: Generated new TSK 0
I20260928 06:28:45.101603 29063 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.225.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab.
Entry for principal kudu/127.22.225.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab.
I20260928 06:28:45.136195 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.129:0
--local_ip_for_outbound_sockets=127.22.225.129
--webserver_interface=127.22.225.129
--webserver_port=0
--tserver_master_addrs=127.22.225.190:38193
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab
--principal=kudu/127.22.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:42431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:28:45.255987 29076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:28:45.256263 29076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:28:45.256297 29076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:28:45.258029 29076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:28:45.258108 29076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:28:45.258152 29076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.129
I20260928 06:28:45.260335 29076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:42431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab
--principal=kudu/127.22.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.225.129
--webserver_port=0
--tserver_master_addrs=127.22.225.190:38193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29076
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.129
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:28:45.260608 29076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:28:45.260842 29076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:28:45.266614 29082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:45.266614 29081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:45.267159 29084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:45.267609 29076 server_base.cc:1034] running on GCE node
I20260928 06:28:45.267903 29076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:28:45.268167 29076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:28:45.269279 29076 hybrid_clock.cc:648] HybridClock initialized: now 1790576925269252 us; error 39 us; skew 500 ppm
Sep 28 06:28:45 dist-test-slave-d59m krb5kdc[28969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576925, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:28:45.271942 29076 init.cc:377] Logged in from keytab as kudu/127.22.225.129@KRBTEST.COM (short username kudu)
I20260928 06:28:45.272405 29076 webserver.cc:460] Webserver started at http://127.22.225.129:41543/ using document root <none> and password file <none>
I20260928 06:28:45.272716 29076 fs_manager.cc:359] Metadata directory not provided
I20260928 06:28:45.272781 29076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:28:45.272917 29076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:28:45.273919 29076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/minicluster-data/ts-0/data/instance:
uuid: "3142050ac29642cbb6a5edb958960f27"
format_stamp: "Formatted at 2026-09-28 06:28:45 on dist-test-slave-d59m"
server_key: "06c57f0bce11751f47bb4e7a069c7756"
server_key_iv: "b901a99654e4aa0b6f74b97b6c156d29"
server_key_version: "encryptionkey@0"
I20260928 06:28:45.274283 29076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/minicluster-data/ts-0/wal/instance:
uuid: "3142050ac29642cbb6a5edb958960f27"
format_stamp: "Formatted at 2026-09-28 06:28:45 on dist-test-slave-d59m"
server_key: "06c57f0bce11751f47bb4e7a069c7756"
server_key_iv: "b901a99654e4aa0b6f74b97b6c156d29"
server_key_version: "encryptionkey@0"
I20260928 06:28:45.275677 29076 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:28:45.276500 29091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:45.276729 29076 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:28:45.276834 29076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/minicluster-data/ts-0/wal
uuid: "3142050ac29642cbb6a5edb958960f27"
format_stamp: "Formatted at 2026-09-28 06:28:45 on dist-test-slave-d59m"
server_key: "06c57f0bce11751f47bb4e7a069c7756"
server_key_iv: "b901a99654e4aa0b6f74b97b6c156d29"
server_key_version: "encryptionkey@0"
I20260928 06:28:45.276923 29076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:28:45.285808 29076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:28:45.288136 29076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:28:45.288300 29076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:28:45.288529 29076 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:28:45.288864 29076 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:28:45.288920 29076 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:45.288959 29076 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:28:45.288986 29076 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:45.295503 29076 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.129:43847
I20260928 06:28:45.295554 29204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.129:43847 every 8 connection(s)
I20260928 06:28:45.295866 29076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
I20260928 06:28:45.302492 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 29076
I20260928 06:28:45.302596 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/minicluster-data/ts-0/wal/instance
I20260928 06:28:45.302766 23430 external_mini_cluster.cc:1428] Setting key 2cef5521e43b5f356d9164502cb65d7c
Sep 28 06:28:45 dist-test-slave-d59m krb5kdc[28969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576925, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.129@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:28:45.307787 29070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:45.296695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:41341 (local address 127.22.225.190:38193)
0928 06:28:45.296762 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:45.296764 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:45.297046 (+ 282us) server_negotiation.cc:407] Connection header received
0928 06:28:45.298412 (+ 1366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:45.298414 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:45.298427 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:45.298503 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:45.299108 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:45.299913 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:45.300958 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:45.301269 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:45.301343 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:45.303766 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:45.303767 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:45.303767 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:45.303775 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:45.306309 (+ 2534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:45.306753 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:45.306755 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:45.306755 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:45.306818 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:45.307111 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:45.307112 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:45.307113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:45.307237 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:45.307316 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:45.307603 (+ 287us) server_negotiation.cc:299] Negotiation successful
0928 06:28:45.307695 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 06:28:45.308003 29207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:45.296690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:38193 (local address 127.22.225.129:41341)
0928 06:28:45.296876 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:45.296930 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:45.298252 (+ 1322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:45.298560 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:45.298563 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:45.298816 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:45.299006 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:45.299010 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:45.300017 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:45.300017 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:45.300879 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:45.300881 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:45.301361 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:45.301362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:45.301459 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:45.301987 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:45.301995 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:45.303661 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:45.306423 (+ 2762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:45.306431 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:45.306434 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:45.306671 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:45.306920 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:45.306922 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:45.306922 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:45.307039 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:45.307308 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:45.307315 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:45.307469 (+ 154us) client_negotiation.cc:769] Sending connection context
0928 06:28:45.307583 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 06:28:45.307748 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":100,"threads_started":1}
I20260928 06:28:45.308600 29205 heartbeater.cc:347] Connected to a master server at 127.22.225.190:38193
I20260928 06:28:45.308700 29205 heartbeater.cc:464] Registering TS with master...
I20260928 06:28:45.308892 29205 heartbeater.cc:511] Master 127.22.225.190:38193 requested a full tablet report, sending...
I20260928 06:28:45.309468 29014 ts_manager.cc:194] Registered new tserver with Master: 3142050ac29642cbb6a5edb958960f27 (127.22.225.129:43847)
I20260928 06:28:45.310082 29014 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.225.129@KRBTEST.COM'} at 127.22.225.129:41341
WARNING: no policy specified for kudu/127.22.225.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab.
Entry for principal kudu/127.22.225.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab.
I20260928 06:28:45.329545 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.130:0
--local_ip_for_outbound_sockets=127.22.225.130
--webserver_interface=127.22.225.130
--webserver_port=0
--tserver_master_addrs=127.22.225.190:38193
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab
--principal=kudu/127.22.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:42431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:28:45.436309 29210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:28:45.436646 29210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:28:45.436707 29210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:28:45.437922 29210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:28:45.437961 29210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:28:45.437986 29210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.130
I20260928 06:28:45.439446 29210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:42431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab
--principal=kudu/127.22.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.225.130
--webserver_port=0
--tserver_master_addrs=127.22.225.190:38193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.130
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:28:45.439690 29210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:28:45.439903 29210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:28:45.444885 29215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:45.444855 29216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:45.445024 29210 server_base.cc:1034] running on GCE node
W20260928 06:28:45.445267 29218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:45.445497 29210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:28:45.445729 29210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:28:45.446882 29210 hybrid_clock.cc:648] HybridClock initialized: now 1790576925446867 us; error 41 us; skew 500 ppm
Sep 28 06:28:45 dist-test-slave-d59m krb5kdc[28969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576925, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:28:45.449440 29210 init.cc:377] Logged in from keytab as kudu/127.22.225.130@KRBTEST.COM (short username kudu)
I20260928 06:28:45.449920 29210 webserver.cc:460] Webserver started at http://127.22.225.130:36789/ using document root <none> and password file <none>
I20260928 06:28:45.450158 29210 fs_manager.cc:359] Metadata directory not provided
I20260928 06:28:45.450223 29210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:28:45.450376 29210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:28:45.451460 29210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/minicluster-data/ts-1/data/instance:
uuid: "0b87032b078b488b819de60990a87015"
format_stamp: "Formatted at 2026-09-28 06:28:45 on dist-test-slave-d59m"
server_key: "a9582fa705e1b948b20df39fc5891ebd"
server_key_iv: "3411b6b9a555e1401bc7a3c93153b2e7"
server_key_version: "encryptionkey@0"
I20260928 06:28:45.451817 29210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/minicluster-data/ts-1/wal/instance:
uuid: "0b87032b078b488b819de60990a87015"
format_stamp: "Formatted at 2026-09-28 06:28:45 on dist-test-slave-d59m"
server_key: "a9582fa705e1b948b20df39fc5891ebd"
server_key_iv: "3411b6b9a555e1401bc7a3c93153b2e7"
server_key_version: "encryptionkey@0"
I20260928 06:28:45.453258 29210 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 06:28:45.454002 29225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:45.454173 29210 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:28:45.454272 29210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/minicluster-data/ts-1/wal
uuid: "0b87032b078b488b819de60990a87015"
format_stamp: "Formatted at 2026-09-28 06:28:45 on dist-test-slave-d59m"
server_key: "a9582fa705e1b948b20df39fc5891ebd"
server_key_iv: "3411b6b9a555e1401bc7a3c93153b2e7"
server_key_version: "encryptionkey@0"
I20260928 06:28:45.454345 29210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:28:45.474516 29210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:28:45.476649 29210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:28:45.476814 29210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:28:45.477047 29210 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:28:45.477355 29210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:28:45.477407 29210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:45.477448 29210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:28:45.477473 29210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:45.483793 29210 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.130:43133
I20260928 06:28:45.483860 29338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.130:43133 every 8 connection(s)
I20260928 06:28:45.484194 29210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
I20260928 06:28:45.485071 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 29210
I20260928 06:28:45.485175 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/minicluster-data/ts-1/wal/instance
I20260928 06:28:45.485375 23430 external_mini_cluster.cc:1428] Setting key 8372058d2fcb93629827d9b5efa33497
Sep 28 06:28:45 dist-test-slave-d59m krb5kdc[28969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576925, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.130@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.225.131@KRBTEST.COM; defaulting to no policy
I20260928 06:28:45.495260 29070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:45.484958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:56315 (local address 127.22.225.190:38193)
0928 06:28:45.485010 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:45.485012 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:45.485295 (+ 283us) server_negotiation.cc:407] Connection header received
0928 06:28:45.486536 (+ 1241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:45.486538 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:45.486551 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:45.486617 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:45.487460 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:45.488146 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:45.489601 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:45.489863 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:45.489901 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:45.492420 (+ 2519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:45.492422 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:45.492422 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:45.492430 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:45.493907 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:45.494283 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:45.494284 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:45.494284 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:45.494349 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:45.494621 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:45.494622 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:45.494622 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:45.494745 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:45.494809 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:45.495093 (+ 284us) server_negotiation.cc:299] Negotiation successful
0928 06:28:45.495172 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:28:45.495404 29341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:45.484966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:38193 (local address 127.22.225.130:56315)
0928 06:28:45.485164 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:45.485210 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:45.486402 (+ 1192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:45.486694 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:45.486698 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:45.487061 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:45.487322 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:45.487327 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:45.488261 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:45.488262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:45.489499 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:45.489502 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:45.489987 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:45.489987 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:45.490112 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:45.490652 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:45.490662 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:45.492303 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:45.493988 (+ 1685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:45.493994 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:45.493996 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:45.494209 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:45.494413 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:45.494414 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:45.494414 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:45.494506 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:45.494808 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:45.494813 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:45.494928 (+ 115us) client_negotiation.cc:769] Sending connection context
0928 06:28:45.495077 (+ 149us) client_negotiation.cc:240] Negotiation successful
0928 06:28:45.495200 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":102,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.131@KRBTEST.COM" created.
I20260928 06:28:45.495857 29339 heartbeater.cc:347] Connected to a master server at 127.22.225.190:38193
I20260928 06:28:45.495965 29339 heartbeater.cc:464] Registering TS with master...
I20260928 06:28:45.496204 29339 heartbeater.cc:511] Master 127.22.225.190:38193 requested a full tablet report, sending...
I20260928 06:28:45.496565 29014 ts_manager.cc:194] Registered new tserver with Master: 0b87032b078b488b819de60990a87015 (127.22.225.130:43133)
I20260928 06:28:45.497107 29014 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.225.130@KRBTEST.COM'} at 127.22.225.130:56315
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab.
Entry for principal kudu/127.22.225.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab.
I20260928 06:28:45.509506 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.131:0
--local_ip_for_outbound_sockets=127.22.225.131
--webserver_interface=127.22.225.131
--webserver_port=0
--tserver_master_addrs=127.22.225.190:38193
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab
--principal=kudu/127.22.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:42431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:28:45.591499 29344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:28:45.591689 29344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:28:45.591710 29344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:28:45.592872 29344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:28:45.592911 29344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:28:45.592948 29344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.131
I20260928 06:28:45.594404 29344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:42431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab
--principal=kudu/127.22.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.225.131
--webserver_port=0
--tserver_master_addrs=127.22.225.190:38193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.131
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:28:45.594637 29344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:28:45.594861 29344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:28:45.599881 29349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:45.599889 29350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:45.600090 29344 server_base.cc:1034] running on GCE node
W20260928 06:28:45.600050 29352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:45.600353 29344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:28:45.600651 29344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:28:45.601815 29344 hybrid_clock.cc:648] HybridClock initialized: now 1790576925601797 us; error 34 us; skew 500 ppm
Sep 28 06:28:45 dist-test-slave-d59m krb5kdc[28969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576925, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:28:45.604451 29344 init.cc:377] Logged in from keytab as kudu/127.22.225.131@KRBTEST.COM (short username kudu)
I20260928 06:28:45.604947 29344 webserver.cc:460] Webserver started at http://127.22.225.131:45497/ using document root <none> and password file <none>
I20260928 06:28:45.605197 29344 fs_manager.cc:359] Metadata directory not provided
I20260928 06:28:45.605253 29344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:28:45.605386 29344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:28:45.606667 29344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/minicluster-data/ts-2/data/instance:
uuid: "0a70e5af9567429b983805981d6483dd"
format_stamp: "Formatted at 2026-09-28 06:28:45 on dist-test-slave-d59m"
server_key: "365fdfef75ff1d46ae2e20df4f1f1275"
server_key_iv: "394f301a061ee4f673eafcb7ef08cff5"
server_key_version: "encryptionkey@0"
I20260928 06:28:45.607046 29344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/minicluster-data/ts-2/wal/instance:
uuid: "0a70e5af9567429b983805981d6483dd"
format_stamp: "Formatted at 2026-09-28 06:28:45 on dist-test-slave-d59m"
server_key: "365fdfef75ff1d46ae2e20df4f1f1275"
server_key_iv: "394f301a061ee4f673eafcb7ef08cff5"
server_key_version: "encryptionkey@0"
I20260928 06:28:45.608615 29344 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260928 06:28:45.609452 29359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:45.609652 29344 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:28:45.609730 29344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/minicluster-data/ts-2/wal
uuid: "0a70e5af9567429b983805981d6483dd"
format_stamp: "Formatted at 2026-09-28 06:28:45 on dist-test-slave-d59m"
server_key: "365fdfef75ff1d46ae2e20df4f1f1275"
server_key_iv: "394f301a061ee4f673eafcb7ef08cff5"
server_key_version: "encryptionkey@0"
I20260928 06:28:45.609793 29344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:28:45.653445 29344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:28:45.655819 29344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:28:45.655952 29344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:28:45.656190 29344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:28:45.656530 29344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:28:45.656594 29344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:45.656626 29344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:28:45.656649 29344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:45.663820 29344 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.131:37579
I20260928 06:28:45.663857 29472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.131:37579 every 8 connection(s)
I20260928 06:28:45.664211 29344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
I20260928 06:28:45.664811 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 29344
I20260928 06:28:45.664894 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/minicluster-data/ts-2/wal/instance
I20260928 06:28:45.665089 23430 external_mini_cluster.cc:1428] Setting key 1c75f5c55fd5376c84040af56535385f
Sep 28 06:28:45 dist-test-slave-d59m krb5kdc[28969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576925, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.131@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:28:45.674991 29070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:45.665243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:48979 (local address 127.22.225.190:38193)
0928 06:28:45.665393 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:45.665396 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:45.665405 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:28:45.666468 (+ 1063us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:45.666469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:45.666478 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:45.666512 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:45.667261 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:45.667727 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:45.669100 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:45.669280 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:45.669333 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:45.672107 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:45.672109 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:45.672109 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:45.672116 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:45.673674 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:45.674121 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:45.674122 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:45.674123 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:45.674176 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:45.674440 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:45.674441 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:45.674441 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:45.674538 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:45.674587 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:45.674867 (+ 280us) server_negotiation.cc:299] Negotiation successful
0928 06:28:45.674929 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260928 06:28:45.675225 29475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:45.664953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:38193 (local address 127.22.225.131:48979)
0928 06:28:45.665154 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:45.665200 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:45.666367 (+ 1167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:45.666590 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:45.666594 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:45.666917 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:45.667180 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:45.667186 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:45.667814 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:45.667815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:45.669023 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:45.669026 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:45.669365 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:45.669366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:45.669508 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:45.670113 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:45.670123 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:45.672005 (+ 1882us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:45.673754 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:45.673760 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:45.673762 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:45.674029 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:45.674245 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:45.674246 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:45.674246 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:45.674377 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:45.674596 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:45.674600 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:45.674744 (+ 144us) client_negotiation.cc:769] Sending connection context
0928 06:28:45.674848 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 06:28:45.674998 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":105,"threads_started":1}
I20260928 06:28:45.675863 29473 heartbeater.cc:347] Connected to a master server at 127.22.225.190:38193
I20260928 06:28:45.675981 29473 heartbeater.cc:464] Registering TS with master...
I20260928 06:28:45.676221 29473 heartbeater.cc:511] Master 127.22.225.190:38193 requested a full tablet report, sending...
I20260928 06:28:45.677583 29014 ts_manager.cc:194] Registered new tserver with Master: 0a70e5af9567429b983805981d6483dd (127.22.225.131:37579)
I20260928 06:28:45.677965 23430 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:28:45.678063 29014 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.225.131@KRBTEST.COM'} at 127.22.225.131:48979
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/krb5kdc/test-admin.keytab.
Sep 28 06:28:45 dist-test-slave-d59m krb5kdc[28969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:28:45.693265 23430 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 06:28:45.694902 29480 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:28:45.694121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38436 (local address 127.22.225.129:43847)
0928 06:28:45.694330 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:45.694335 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:45.694355 (+ 20us) server_negotiation.cc:407] Connection header received
0928 06:28:45.694386 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:45.694388 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:45.694406 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:45.694487 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:45.694822 (+ 335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38436: BlockingRecv error: recv got EOF from 127.0.0.1:38436 (error 108)
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":101,"threads_started":1}
W20260928 06:28:45.694957 23430 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/krb5kdc/krb5cc' of type FILE
Sep 28 06:28:45 dist-test-slave-d59m krb5kdc[28969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:28:45.696442 23430 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 06:28:45 dist-test-slave-d59m krb5kdc[28969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:28:45.697402 23430 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 06:28:45 dist-test-slave-d59m krb5kdc[28969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.225.129@KRBTEST.COM
I20260928 06:28:45.705335 29480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:45.697910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38452 (local address 127.22.225.129:43847)
0928 06:28:45.697954 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:45.697956 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:45.698000 (+ 44us) server_negotiation.cc:407] Connection header received
0928 06:28:45.698070 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:45.698071 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:45.698082 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:45.698153 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:45.698596 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:45.699308 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:45.700055 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:45.700364 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:45.700445 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:45.702544 (+ 2099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:45.702552 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:45.702552 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:45.702566 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:45.704123 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:45.704497 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:45.704501 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:45.704501 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:45.704564 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:45.704797 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:45.704798 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:45.704799 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:45.704934 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:45.705091 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:45.705140 (+ 49us) server_negotiation.cc:299] Negotiation successful
0928 06:28:45.705221 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
W20260928 06:28:45.707146 29480 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:28:45.706624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38460 (local address 127.22.225.129:43847)
0928 06:28:45.706668 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:45.706670 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:45.706682 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:28:45.706754 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:45.706755 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:45.706766 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:45.706814 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:45.707107 (+ 293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38460: BlockingRecv error: recv got EOF from 127.0.0.1:38460 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 06:28:45.707252 23430 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/krb5kdc/krb5cc)
Sep 28 06:28:45 dist-test-slave-d59m krb5kdc[28969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:28:45.708272 23430 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 06:28:45 dist-test-slave-d59m krb5kdc[28969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:28:45.709178 23430 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 06:28:45 dist-test-slave-d59m krb5kdc[28969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.225.129@KRBTEST.COM
I20260928 06:28:45.716248 29480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:45.709711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38462 (local address 127.22.225.129:43847)
0928 06:28:45.709752 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:45.709754 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:45.709794 (+ 40us) server_negotiation.cc:407] Connection header received
0928 06:28:45.709864 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:45.709865 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:45.709875 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:45.709946 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:45.710303 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:45.710767 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:45.711434 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:45.711632 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:45.711705 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:45.713794 (+ 2089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:45.713795 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:45.713795 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:45.713803 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:45.715162 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:45.715494 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:45.715495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:45.715496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:45.715556 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:45.715762 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:45.715763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:45.715763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:45.715861 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:45.715957 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:45.716076 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 06:28:45.716154 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 06:28:45.717867 29480 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:28:45.717253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38470 (local address 127.22.225.129:43847)
0928 06:28:45.717293 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:45.717295 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:45.717384 (+ 89us) server_negotiation.cc:407] Connection header received
0928 06:28:45.717409 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:45.717409 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:45.717419 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:45.717485 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:45.717825 (+ 340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38470: BlockingRecv error: recv got EOF from 127.0.0.1:38470 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 06:28:45.717916 23430 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576899143097-23430-0/krb5kdc/krb5cc' of type FILE
Sep 28 06:28:45 dist-test-slave-d59m krb5kdc[28969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:28:45.718897 23430 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 06:28:45 dist-test-slave-d59m krb5kdc[28969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:28:45.719760 23430 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 06:28:45 dist-test-slave-d59m krb5kdc[28969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.225.129@KRBTEST.COM
I20260928 06:28:45.726877 29480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:45.720252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38472 (local address 127.22.225.129:43847)
0928 06:28:45.720312 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:45.720314 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:45.720326 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:28:45.720386 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:45.720387 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:45.720396 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:45.720454 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:45.720862 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:45.721562 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:45.722201 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:45.722471 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:45.722543 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:45.724568 (+ 2025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:45.724569 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:45.724569 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:45.724607 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:45.725898 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:45.726253 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:45.726253 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:45.726254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:45.726300 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:45.726491 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:45.726491 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:45.726492 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:45.726580 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:45.726647 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:45.726770 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 06:28:45.726819 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 06:28:45.727506 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 29076
I20260928 06:28:45.733387 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 29210
I20260928 06:28:45.738338 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 29344
I20260928 06:28:45.742686 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 28982
2026-09-28T06:28:45Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1987 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:28:45 dist-test-slave-d59m krb5kdc[29493](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:28:45 dist-test-slave-d59m krb5kdc[29493](info): set up 2 sockets
Sep 28 06:28:45 dist-test-slave-d59m krb5kdc[29493](info): commencing operation
krb5kdc: starting...
W20260928 06:28:46.826850 23430 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Sep 28 06:28:46 dist-test-slave-d59m krb5kdc[29493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:28:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:28:46Z Disabled control of system clock
WARNING: no policy specified for oryx/127.22.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/krb5kdc/oryx_127.22.225.190.keytab.
Entry for principal oryx/127.22.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/krb5kdc/oryx_127.22.225.190.keytab.
I20260928 06:28:46.947536 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:41857
--webserver_interface=127.22.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/krb5kdc/oryx_127.22.225.190.keytab
--principal=oryx/127.22.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:36117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.225.190:41857
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:28:47.029246 29506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:28:47.029460 29506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:28:47.029490 29506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:28:47.030668 29506 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:28:47.030725 29506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:28:47.030772 29506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:28:47.030791 29506 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:28:47.030817 29506 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:28:47.032505 29506 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:36117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.225.190:41857
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/krb5kdc/oryx_127.22.225.190.keytab
--principal=oryx/127.22.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:41857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:28:47.032804 29506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:28:47.033049 29506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:28:47.037961 29511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:47.038170 29506 server_base.cc:1034] running on GCE node
W20260928 06:28:47.037966 29512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:47.037971 29514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:47.038735 29506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:28:47.039059 29506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:28:47.040228 29506 hybrid_clock.cc:648] HybridClock initialized: now 1790576927040201 us; error 44 us; skew 500 ppm
Sep 28 06:28:47 dist-test-slave-d59m krb5kdc[29493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576927, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:28:47.042899 29506 init.cc:377] Logged in from keytab as oryx/127.22.225.190@KRBTEST.COM (short username oryx)
I20260928 06:28:47.043404 29506 webserver.cc:460] Webserver started at http://127.22.225.190:36103/ using document root <none> and password file <none>
I20260928 06:28:47.043663 29506 fs_manager.cc:359] Metadata directory not provided
I20260928 06:28:47.043732 29506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:28:47.043888 29506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:28:47.045053 29506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/master-0/data/instance:
uuid: "0c07c7b5cb55443299f41d25e3dc513d"
format_stamp: "Formatted at 2026-09-28 06:28:47 on dist-test-slave-d59m"
server_key: "00181806bc6aaf668314cd547eeecb11"
server_key_iv: "640cd126d3acd5a9a5bef032fd466464"
server_key_version: "encryptionkey@0"
I20260928 06:28:47.045403 29506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/master-0/wal/instance:
uuid: "0c07c7b5cb55443299f41d25e3dc513d"
format_stamp: "Formatted at 2026-09-28 06:28:47 on dist-test-slave-d59m"
server_key: "00181806bc6aaf668314cd547eeecb11"
server_key_iv: "640cd126d3acd5a9a5bef032fd466464"
server_key_version: "encryptionkey@0"
I20260928 06:28:47.046799 29506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 06:28:47.047586 29521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:47.047765 29506 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:28:47.047870 29506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/master-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/master-0/wal
uuid: "0c07c7b5cb55443299f41d25e3dc513d"
format_stamp: "Formatted at 2026-09-28 06:28:47 on dist-test-slave-d59m"
server_key: "00181806bc6aaf668314cd547eeecb11"
server_key_iv: "640cd126d3acd5a9a5bef032fd466464"
server_key_version: "encryptionkey@0"
I20260928 06:28:47.047945 29506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:28:47.070042 29506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:28:47.072220 29506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:28:47.072384 29506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:28:47.076860 29506 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.190:41857
I20260928 06:28:47.076886 29573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.190:41857 every 8 connection(s)
I20260928 06:28:47.077260 29506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
I20260928 06:28:47.078294 29574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:47.080559 29574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c07c7b5cb55443299f41d25e3dc513d: Bootstrap starting.
I20260928 06:28:47.081219 29574 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c07c7b5cb55443299f41d25e3dc513d: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:47.081516 29574 log.cc:824] T 00000000000000000000000000000000 P 0c07c7b5cb55443299f41d25e3dc513d: Log is configured to *not* fsync() on all Append() calls
I20260928 06:28:47.082267 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 29506
I20260928 06:28:47.082379 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/master-0/wal/instance
I20260928 06:28:47.082377 29574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c07c7b5cb55443299f41d25e3dc513d: No bootstrap required, opened a new log
I20260928 06:28:47.082576 23430 external_mini_cluster.cc:1428] Setting key 2a32322c9640854ca93ee77e54c4e13b
I20260928 06:28:47.084229 29574 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c07c7b5cb55443299f41d25e3dc513d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c07c7b5cb55443299f41d25e3dc513d" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 41857 } }
I20260928 06:28:47.084367 29574 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c07c7b5cb55443299f41d25e3dc513d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:47.084389 29574 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c07c7b5cb55443299f41d25e3dc513d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c07c7b5cb55443299f41d25e3dc513d, State: Initialized, Role: FOLLOWER
I20260928 06:28:47.084537 29574 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c07c7b5cb55443299f41d25e3dc513d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c07c7b5cb55443299f41d25e3dc513d" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 41857 } }
I20260928 06:28:47.084683 29574 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c07c7b5cb55443299f41d25e3dc513d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:28:47.084726 29574 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c07c7b5cb55443299f41d25e3dc513d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:28:47.084770 29574 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c07c7b5cb55443299f41d25e3dc513d [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:47.085630 29574 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c07c7b5cb55443299f41d25e3dc513d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c07c7b5cb55443299f41d25e3dc513d" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 41857 } }
I20260928 06:28:47.085750 29574 leader_election.cc:302] T 00000000000000000000000000000000 P 0c07c7b5cb55443299f41d25e3dc513d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c07c7b5cb55443299f41d25e3dc513d; no voters:
I20260928 06:28:47.085938 29574 leader_election.cc:288] T 00000000000000000000000000000000 P 0c07c7b5cb55443299f41d25e3dc513d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:28:47.085983 29579 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c07c7b5cb55443299f41d25e3dc513d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:28:47.086127 29579 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c07c7b5cb55443299f41d25e3dc513d [term 1 LEADER]: Becoming Leader. State: Replica: 0c07c7b5cb55443299f41d25e3dc513d, State: Running, Role: LEADER
I20260928 06:28:47.086186 29574 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c07c7b5cb55443299f41d25e3dc513d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:28:47.086205 29579 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c07c7b5cb55443299f41d25e3dc513d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c07c7b5cb55443299f41d25e3dc513d" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 41857 } }
I20260928 06:28:47.086519 29579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c07c7b5cb55443299f41d25e3dc513d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c07c7b5cb55443299f41d25e3dc513d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c07c7b5cb55443299f41d25e3dc513d" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 41857 } } }
I20260928 06:28:47.086575 29579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c07c7b5cb55443299f41d25e3dc513d [sys.catalog]: This master's current role is: LEADER
I20260928 06:28:47.086678 29579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c07c7b5cb55443299f41d25e3dc513d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c07c7b5cb55443299f41d25e3dc513d. Latest consensus state: current_term: 1 leader_uuid: "0c07c7b5cb55443299f41d25e3dc513d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c07c7b5cb55443299f41d25e3dc513d" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 41857 } } }
I20260928 06:28:47.086712 29579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c07c7b5cb55443299f41d25e3dc513d [sys.catalog]: This master's current role is: LEADER
I20260928 06:28:47.086818 29584 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:28:47.087284 29584 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:28:47.089610 29596 mvcc.cc:205] Tried to move back new op lower bound from 7334203093357187072 to 7334203093345456128. Current Snapshot: MvccSnapshot[applied={T|T < 7334203093357187072}]
I20260928 06:28:47.090189 29584 catalog_manager.cc:1347] Generated new cluster ID: 3fb00626f39f44199dbf917853eb1818
I20260928 06:28:47.090243 29584 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:28:47 dist-test-slave-d59m krb5kdc[29493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM
I20260928 06:28:47.100696 29578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:47.083135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53698 (local address 127.22.225.190:41857)
0928 06:28:47.083345 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:47.083355 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:47.084658 (+ 1303us) server_negotiation.cc:407] Connection header received
0928 06:28:47.084885 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:47.084892 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:47.084948 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:47.085062 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:47.085938 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:47.087459 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:47.088772 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:47.089350 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:47.089462 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:47.092440 (+ 2978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:47.092444 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:47.092444 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:47.092457 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:47.098476 (+ 6019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:47.098926 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:47.098929 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:47.098929 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:47.099056 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:47.099259 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:47.099260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:47.099260 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:47.099575 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:47.100186 (+ 611us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:47.100262 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 06:28:47.100408 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":102,"threads_started":1}
I20260928 06:28:47.101331 29584 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:28:47.101948 29584 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:28:47.114575 29584 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c07c7b5cb55443299f41d25e3dc513d: Generated new TSK 0
I20260928 06:28:47.114809 29584 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.22.225.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.225.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.225.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/krb5kdc/oryx_127.22.225.129.keytab.
Entry for principal oryx/127.22.225.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/krb5kdc/oryx_127.22.225.129.keytab.
I20260928 06:28:47.148165 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.129:0
--local_ip_for_outbound_sockets=127.22.225.129
--webserver_interface=127.22.225.129
--webserver_port=0
--tserver_master_addrs=127.22.225.190:41857
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/krb5kdc/oryx_127.22.225.129.keytab
--principal=oryx/127.22.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:36117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:28:47.265831 29600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:28:47.266077 29600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:28:47.266119 29600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:28:47.267262 29600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:28:47.267331 29600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:28:47.267388 29600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.129
I20260928 06:28:47.268865 29600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:36117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/krb5kdc/oryx_127.22.225.129.keytab
--principal=oryx/127.22.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.225.129
--webserver_port=0
--tserver_master_addrs=127.22.225.190:41857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.129
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:28:47.269125 29600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:28:47.269352 29600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:28:47.274461 29606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:47.274675 29600 server_base.cc:1034] running on GCE node
W20260928 06:28:47.274761 29608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:47.274792 29605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:47.275192 29600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:28:47.275512 29600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:28:47.276705 29600 hybrid_clock.cc:648] HybridClock initialized: now 1790576927276682 us; error 37 us; skew 500 ppm
Sep 28 06:28:47 dist-test-slave-d59m krb5kdc[29493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576927, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:28:47.279386 29600 init.cc:377] Logged in from keytab as oryx/127.22.225.129@KRBTEST.COM (short username oryx)
I20260928 06:28:47.279906 29600 webserver.cc:460] Webserver started at http://127.22.225.129:40407/ using document root <none> and password file <none>
I20260928 06:28:47.280144 29600 fs_manager.cc:359] Metadata directory not provided
I20260928 06:28:47.280189 29600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:28:47.280315 29600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:28:47.281455 29600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-0/data/instance:
uuid: "c1813e3cd67549a1a161c3c54abc3c3b"
format_stamp: "Formatted at 2026-09-28 06:28:47 on dist-test-slave-d59m"
server_key: "3d6130a81a4088bf3f7160dfbdb64c6d"
server_key_iv: "569e482679cd3450c0a2176c9b856ed3"
server_key_version: "encryptionkey@0"
I20260928 06:28:47.281812 29600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-0/wal/instance:
uuid: "c1813e3cd67549a1a161c3c54abc3c3b"
format_stamp: "Formatted at 2026-09-28 06:28:47 on dist-test-slave-d59m"
server_key: "3d6130a81a4088bf3f7160dfbdb64c6d"
server_key_iv: "569e482679cd3450c0a2176c9b856ed3"
server_key_version: "encryptionkey@0"
I20260928 06:28:47.283169 29600 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 06:28:47.283972 29615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:47.284184 29600 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:28:47.284288 29600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-0/wal
uuid: "c1813e3cd67549a1a161c3c54abc3c3b"
format_stamp: "Formatted at 2026-09-28 06:28:47 on dist-test-slave-d59m"
server_key: "3d6130a81a4088bf3f7160dfbdb64c6d"
server_key_iv: "569e482679cd3450c0a2176c9b856ed3"
server_key_version: "encryptionkey@0"
I20260928 06:28:47.284369 29600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:28:47.297694 29600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:28:47.300032 29600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:28:47.300194 29600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:28:47.300431 29600 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:28:47.300812 29600 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:28:47.300873 29600 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:47.300915 29600 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:28:47.300942 29600 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:47.306650 29600 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.129:40749
I20260928 06:28:47.306770 29728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.129:40749 every 8 connection(s)
I20260928 06:28:47.307070 29600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
Sep 28 06:28:47 dist-test-slave-d59m krb5kdc[29493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576927, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.225.129@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM
I20260928 06:28:47.314904 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 29600
I20260928 06:28:47.315006 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-0/wal/instance
I20260928 06:28:47.315210 23430 external_mini_cluster.cc:1428] Setting key 174b1a82306aa295155b4af5979c6647
I20260928 06:28:47.318629 29578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:47.307834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:35091 (local address 127.22.225.190:41857)
0928 06:28:47.307912 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:47.307914 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:47.308163 (+ 249us) server_negotiation.cc:407] Connection header received
0928 06:28:47.308884 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:47.308885 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:47.308899 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:47.308940 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:47.310247 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:47.310742 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:47.312325 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:47.312513 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:47.312563 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:47.315093 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:47.315094 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:47.315095 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:47.315102 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:47.316975 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:47.317508 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:47.317510 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:47.317510 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:47.317576 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:47.317862 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:47.317864 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:47.317864 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:47.318029 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:47.318108 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:47.318468 (+ 360us) server_negotiation.cc:299] Negotiation successful
0928 06:28:47.318560 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 06:28:47.318897 29731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:47.307797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:41857 (local address 127.22.225.129:35091)
0928 06:28:47.308008 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:47.308061 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:47.308774 (+ 713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:47.309032 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:47.309036 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:47.309840 (+ 804us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:47.310150 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:47.310157 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:47.310834 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:47.310835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:47.312227 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:47.312230 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:47.312625 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:47.312626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:47.312772 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:47.313305 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:47.313316 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:47.314943 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:47.317063 (+ 2120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:47.317071 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:47.317073 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:47.317401 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:47.317653 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:47.317654 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:47.317654 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:47.317780 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:47.318113 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:47.318121 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:47.318297 (+ 176us) client_negotiation.cc:769] Sending connection context
0928 06:28:47.318425 (+ 128us) client_negotiation.cc:240] Negotiation successful
0928 06:28:47.318622 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":115,"threads_started":1}
I20260928 06:28:47.319360 29729 heartbeater.cc:347] Connected to a master server at 127.22.225.190:41857
I20260928 06:28:47.319485 29729 heartbeater.cc:464] Registering TS with master...
I20260928 06:28:47.319677 29729 heartbeater.cc:511] Master 127.22.225.190:41857 requested a full tablet report, sending...
I20260928 06:28:47.320245 29538 ts_manager.cc:194] Registered new tserver with Master: c1813e3cd67549a1a161c3c54abc3c3b (127.22.225.129:40749)
I20260928 06:28:47.320998 29538 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.225.129@KRBTEST.COM'} at 127.22.225.129:35091
WARNING: no policy specified for oryx/127.22.225.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.225.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.225.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/krb5kdc/oryx_127.22.225.130.keytab.
Entry for principal oryx/127.22.225.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/krb5kdc/oryx_127.22.225.130.keytab.
I20260928 06:28:47.343544 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.130:0
--local_ip_for_outbound_sockets=127.22.225.130
--webserver_interface=127.22.225.130
--webserver_port=0
--tserver_master_addrs=127.22.225.190:41857
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/krb5kdc/oryx_127.22.225.130.keytab
--principal=oryx/127.22.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:36117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:28:47.432600 29734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:28:47.432811 29734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:28:47.432833 29734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:28:47.433938 29734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:28:47.433976 29734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:28:47.434001 29734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.130
I20260928 06:28:47.435546 29734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:36117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/krb5kdc/oryx_127.22.225.130.keytab
--principal=oryx/127.22.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.225.130
--webserver_port=0
--tserver_master_addrs=127.22.225.190:41857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.130
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:28:47.435788 29734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:28:47.436019 29734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:28:47.441737 29739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:47.441735 29740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:47.441955 29734 server_base.cc:1034] running on GCE node
W20260928 06:28:47.441984 29742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:47.442237 29734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:28:47.442473 29734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:28:47.443650 29734 hybrid_clock.cc:648] HybridClock initialized: now 1790576927443635 us; error 31 us; skew 500 ppm
Sep 28 06:28:47 dist-test-slave-d59m krb5kdc[29493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576927, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:28:47.446192 29734 init.cc:377] Logged in from keytab as oryx/127.22.225.130@KRBTEST.COM (short username oryx)
I20260928 06:28:47.446630 29734 webserver.cc:460] Webserver started at http://127.22.225.130:36513/ using document root <none> and password file <none>
I20260928 06:28:47.446859 29734 fs_manager.cc:359] Metadata directory not provided
I20260928 06:28:47.446933 29734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:28:47.447074 29734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:28:47.448076 29734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-1/data/instance:
uuid: "3a7683a995ae4403b291afc021e4e0d3"
format_stamp: "Formatted at 2026-09-28 06:28:47 on dist-test-slave-d59m"
server_key: "9129e90b6ac80f7cd44caeb89a51fcd7"
server_key_iv: "806da175cb7892f04dce3356107b8449"
server_key_version: "encryptionkey@0"
I20260928 06:28:47.448427 29734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-1/wal/instance:
uuid: "3a7683a995ae4403b291afc021e4e0d3"
format_stamp: "Formatted at 2026-09-28 06:28:47 on dist-test-slave-d59m"
server_key: "9129e90b6ac80f7cd44caeb89a51fcd7"
server_key_iv: "806da175cb7892f04dce3356107b8449"
server_key_version: "encryptionkey@0"
I20260928 06:28:47.449908 29734 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:28:47.450764 29749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:47.450958 29734 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:28:47.451051 29734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-1/wal
uuid: "3a7683a995ae4403b291afc021e4e0d3"
format_stamp: "Formatted at 2026-09-28 06:28:47 on dist-test-slave-d59m"
server_key: "9129e90b6ac80f7cd44caeb89a51fcd7"
server_key_iv: "806da175cb7892f04dce3356107b8449"
server_key_version: "encryptionkey@0"
I20260928 06:28:47.451134 29734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:28:47.465931 29734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:28:47.468102 29734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:28:47.468256 29734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:28:47.468487 29734 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:28:47.468972 29734 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:28:47.469010 29734 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:47.469033 29734 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:28:47.469045 29734 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:47.476151 29734 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.130:43647
I20260928 06:28:47.476191 29862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.130:43647 every 8 connection(s)
I20260928 06:28:47.476516 29734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
I20260928 06:28:47.478870 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 29734
I20260928 06:28:47.478967 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-1/wal/instance
I20260928 06:28:47.479141 23430 external_mini_cluster.cc:1428] Setting key bb03c32140e22556fe668492b07bd6fd
Sep 28 06:28:47 dist-test-slave-d59m krb5kdc[29493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576927, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.225.130@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM
WARNING: no policy specified for oryx/127.22.225.131@KRBTEST.COM; defaulting to no policy
I20260928 06:28:47.488083 29578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:47.477320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:52227 (local address 127.22.225.190:41857)
0928 06:28:47.477365 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:47.477367 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:47.477582 (+ 215us) server_negotiation.cc:407] Connection header received
0928 06:28:47.478099 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:47.478100 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:47.478113 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:47.478190 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:47.479347 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:47.480044 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:47.481414 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:47.481722 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:47.481802 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:47.484627 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:47.484628 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:47.484629 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:47.484637 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:47.486354 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:47.486865 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:47.486867 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:47.486867 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:47.486928 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:47.487252 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:47.487253 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:47.487253 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:47.487391 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:47.487457 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:47.487843 (+ 386us) server_negotiation.cc:299] Negotiation successful
0928 06:28:47.487923 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:28:47.488283 29865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:47.477304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:41857 (local address 127.22.225.130:52227)
0928 06:28:47.477472 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:47.477507 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:47.477992 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:47.478240 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:47.478243 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:47.478968 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:47.479260 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:47.479266 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:47.480135 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:47.480136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:47.481323 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:47.481326 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:47.481816 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:47.481817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:47.481966 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:47.482586 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:47.482596 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:47.484506 (+ 1910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:47.486450 (+ 1944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:47.486457 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:47.486459 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:47.486774 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:47.487002 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:47.487003 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:47.487004 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:47.487146 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:47.487517 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:47.487525 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:47.487695 (+ 170us) client_negotiation.cc:769] Sending connection context
0928 06:28:47.487814 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 06:28:47.487986 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":92,"threads_started":1}
I20260928 06:28:47.488947 29863 heartbeater.cc:347] Connected to a master server at 127.22.225.190:41857
I20260928 06:28:47.489050 29863 heartbeater.cc:464] Registering TS with master...
I20260928 06:28:47.489234 29863 heartbeater.cc:511] Master 127.22.225.190:41857 requested a full tablet report, sending...
I20260928 06:28:47.489632 29538 ts_manager.cc:194] Registered new tserver with Master: 3a7683a995ae4403b291afc021e4e0d3 (127.22.225.130:43647)
I20260928 06:28:47.490073 29538 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.225.130@KRBTEST.COM'} at 127.22.225.130:52227
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.225.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.225.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/krb5kdc/oryx_127.22.225.131.keytab.
Entry for principal oryx/127.22.225.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/krb5kdc/oryx_127.22.225.131.keytab.
I20260928 06:28:47.505477 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.131:0
--local_ip_for_outbound_sockets=127.22.225.131
--webserver_interface=127.22.225.131
--webserver_port=0
--tserver_master_addrs=127.22.225.190:41857
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/krb5kdc/oryx_127.22.225.131.keytab
--principal=oryx/127.22.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:36117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:28:47.596375 29868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:28:47.596618 29868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:28:47.596657 29868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:28:47.597765 29868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:28:47.597829 29868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:28:47.597873 29868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.131
I20260928 06:28:47.599315 29868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:36117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/krb5kdc/oryx_127.22.225.131.keytab
--principal=oryx/127.22.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.225.131
--webserver_port=0
--tserver_master_addrs=127.22.225.190:41857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.131
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:28:47.599584 29868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:28:47.599815 29868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:28:47.604969 29873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:47.604951 29874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:47.604941 29876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:47.605242 29868 server_base.cc:1034] running on GCE node
I20260928 06:28:47.605412 29868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:28:47.605629 29868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:28:47.606781 29868 hybrid_clock.cc:648] HybridClock initialized: now 1790576927606763 us; error 31 us; skew 500 ppm
Sep 28 06:28:47 dist-test-slave-d59m krb5kdc[29493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576927, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:28:47.609534 29868 init.cc:377] Logged in from keytab as oryx/127.22.225.131@KRBTEST.COM (short username oryx)
I20260928 06:28:47.609992 29868 webserver.cc:460] Webserver started at http://127.22.225.131:37481/ using document root <none> and password file <none>
I20260928 06:28:47.610256 29868 fs_manager.cc:359] Metadata directory not provided
I20260928 06:28:47.610301 29868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:28:47.610441 29868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:28:47.611395 29868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-2/data/instance:
uuid: "377df16ac26d45a8b8114cc6d81d4894"
format_stamp: "Formatted at 2026-09-28 06:28:47 on dist-test-slave-d59m"
server_key: "adc62e2666ea8a27a7999655e1e533e0"
server_key_iv: "860d90871f7420df1624b7d756cb49f9"
server_key_version: "encryptionkey@0"
I20260928 06:28:47.611732 29868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-2/wal/instance:
uuid: "377df16ac26d45a8b8114cc6d81d4894"
format_stamp: "Formatted at 2026-09-28 06:28:47 on dist-test-slave-d59m"
server_key: "adc62e2666ea8a27a7999655e1e533e0"
server_key_iv: "860d90871f7420df1624b7d756cb49f9"
server_key_version: "encryptionkey@0"
I20260928 06:28:47.613119 29868 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:28:47.613837 29883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:47.614012 29868 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:28:47.614107 29868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-2/wal
uuid: "377df16ac26d45a8b8114cc6d81d4894"
format_stamp: "Formatted at 2026-09-28 06:28:47 on dist-test-slave-d59m"
server_key: "adc62e2666ea8a27a7999655e1e533e0"
server_key_iv: "860d90871f7420df1624b7d756cb49f9"
server_key_version: "encryptionkey@0"
I20260928 06:28:47.614180 29868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:28:47.627969 29868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:28:47.630437 29868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:28:47.630590 29868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:28:47.630803 29868 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:28:47.631131 29868 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:28:47.631193 29868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:47.631235 29868 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:28:47.631261 29868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:47.638392 29868 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.131:38727
I20260928 06:28:47.638454 29996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.131:38727 every 8 connection(s)
I20260928 06:28:47.638803 29868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
I20260928 06:28:47.640543 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 29868
I20260928 06:28:47.640674 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-2/wal/instance
I20260928 06:28:47.640823 23430 external_mini_cluster.cc:1428] Setting key 87ec040c4cc0a00d8db3bc7fcbcf19ca
Sep 28 06:28:47 dist-test-slave-d59m krb5kdc[29493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576927, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.225.131@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM
I20260928 06:28:47.648712 29578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:47.639572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:51551 (local address 127.22.225.190:41857)
0928 06:28:47.639643 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:47.639646 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:47.639902 (+ 256us) server_negotiation.cc:407] Connection header received
0928 06:28:47.640497 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:47.640498 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:47.640512 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:47.640617 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:47.641692 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:47.642162 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:47.643391 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:47.643561 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:47.643608 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:47.645924 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:47.645925 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:47.645925 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:47.645933 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:47.647339 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:47.647733 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:47.647734 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:47.647735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:47.647794 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:47.648038 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:47.648039 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:47.648040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:47.648153 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:47.648228 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:47.648501 (+ 273us) server_negotiation.cc:299] Negotiation successful
0928 06:28:47.648622 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 06:28:47.648896 29999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:47.639558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:41857 (local address 127.22.225.131:51551)
0928 06:28:47.639760 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:47.639805 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:47.640391 (+ 586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:47.640666 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:47.640669 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:47.641372 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:47.641613 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:47.641619 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:47.642242 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:47.642243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:47.643312 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:47.643315 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:47.643633 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:47.643634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:47.643758 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:47.644289 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:47.644298 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:47.645783 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:47.647423 (+ 1640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:47.647429 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:47.647432 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:47.647657 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:47.647861 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:47.647862 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:47.647862 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:47.647972 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:47.648219 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:47.648225 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:47.648362 (+ 137us) client_negotiation.cc:769] Sending connection context
0928 06:28:47.648462 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 06:28:47.648670 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":102,"threads_started":1}
I20260928 06:28:47.649382 29997 heartbeater.cc:347] Connected to a master server at 127.22.225.190:41857
I20260928 06:28:47.649493 29997 heartbeater.cc:464] Registering TS with master...
I20260928 06:28:47.649667 29997 heartbeater.cc:511] Master 127.22.225.190:41857 requested a full tablet report, sending...
I20260928 06:28:47.650044 29538 ts_manager.cc:194] Registered new tserver with Master: 377df16ac26d45a8b8114cc6d81d4894 (127.22.225.131:38727)
I20260928 06:28:47.650405 29538 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.225.131@KRBTEST.COM'} at 127.22.225.131:51551
I20260928 06:28:47.652264 23430 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.22.225.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.225.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.225.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/krb5kdc/oryx_127.22.225.189.keytab.
Entry for principal oryx/127.22.225.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/krb5kdc/oryx_127.22.225.189.keytab.
I20260928 06:28:47.678682 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.189:35431
--webserver_interface=127.22.225.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/krb5kdc/oryx_127.22.225.189.keytab
--principal=oryx/127.22.225.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:36117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.225.190:41857,127.22.225.189:35431
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:28:47.760257 30002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:28:47.760494 30002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:28:47.760540 30002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:28:47.761745 30002 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:28:47.761814 30002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:28:47.761837 30002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:28:47.761858 30002 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:28:47.761889 30002 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:28:47.763417 30002 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:36117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.22.225.190:41857,127.22.225.189:35431
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/krb5kdc/oryx_127.22.225.189.keytab
--principal=oryx/127.22.225.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.189:35431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.22.225.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
W20260928 06:28:47.763577 30002 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.22.225.190:41857,127.22.225.189:35431'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 06:28:47.763733 30002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:28:47.763949 30002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:28:47.768847 30010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:47.768870 30008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:47.768847 30007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:47.769270 30002 server_base.cc:1034] running on GCE node
I20260928 06:28:47.769524 30002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:28:47.769831 30002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:28:47.770991 30002 hybrid_clock.cc:648] HybridClock initialized: now 1790576927770981 us; error 26 us; skew 500 ppm
Sep 28 06:28:47 dist-test-slave-d59m krb5kdc[29493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576927, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.225.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:28:47.773743 30002 init.cc:377] Logged in from keytab as oryx/127.22.225.189@KRBTEST.COM (short username oryx)
I20260928 06:28:47.774263 30002 webserver.cc:460] Webserver started at http://127.22.225.189:41769/ using document root <none> and password file <none>
I20260928 06:28:47.774524 30002 fs_manager.cc:359] Metadata directory not provided
I20260928 06:28:47.774592 30002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:28:47.774722 30002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:28:47.776090 30002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/master-1/data/instance:
uuid: "f15b6ec81d284f2baa79fa54469b144b"
format_stamp: "Formatted at 2026-09-28 06:28:47 on dist-test-slave-d59m"
server_key: "8d65bb6f3c2eef57dceba1167d7cb667"
server_key_iv: "106153f3750ee2bc0e0460965f2185c2"
server_key_version: "encryptionkey@0"
I20260928 06:28:47.776517 30002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/master-1/wal/instance:
uuid: "f15b6ec81d284f2baa79fa54469b144b"
format_stamp: "Formatted at 2026-09-28 06:28:47 on dist-test-slave-d59m"
server_key: "8d65bb6f3c2eef57dceba1167d7cb667"
server_key_iv: "106153f3750ee2bc0e0460965f2185c2"
server_key_version: "encryptionkey@0"
I20260928 06:28:47.778192 30002 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:28:47.779029 30017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:47.779233 30002 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:28:47.779326 30002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/master-1/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/master-1/wal
uuid: "f15b6ec81d284f2baa79fa54469b144b"
format_stamp: "Formatted at 2026-09-28 06:28:47 on dist-test-slave-d59m"
server_key: "8d65bb6f3c2eef57dceba1167d7cb667"
server_key_iv: "106153f3750ee2bc0e0460965f2185c2"
server_key_version: "encryptionkey@0"
I20260928 06:28:47.779397 30002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:28:47.799939 30002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:28:47.802567 30002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:28:47.802735 30002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:28:47.807482 30002 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.189:35431
I20260928 06:28:47.807509 30069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.189:35431 every 8 connection(s)
I20260928 06:28:47.807891 30002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/master-1/data/info.pb
I20260928 06:28:47.808986 30070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:47.810606 30070 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.225.190" port: 41857 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:28:47.813630 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 30002
Sep 28 06:28:47 dist-test-slave-d59m krb5kdc[29493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576927, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.225.189@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM
Sep 28 06:28:47 dist-test-slave-d59m krb5kdc[29493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.225.189@KRBTEST.COM
I20260928 06:28:47.821938 29578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:47.811396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53712 (local address 127.22.225.190:41857)
0928 06:28:47.811455 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:47.811457 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:47.811678 (+ 221us) server_negotiation.cc:407] Connection header received
0928 06:28:47.812172 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:47.812173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:47.812186 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:47.812251 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:47.812968 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:47.813471 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:47.814911 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:47.815120 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:47.815205 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:47.817809 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:47.817811 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:47.817812 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:47.817819 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:47.820299 (+ 2480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:47.820855 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:47.820857 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:47.820857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:47.820920 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:47.821189 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:47.821190 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:47.821191 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:47.821387 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:47.821461 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:47.821802 (+ 341us) server_negotiation.cc:299] Negotiation successful
0928 06:28:47.821869 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 06:28:47.822212 30072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:47.811371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:41857 (local address 127.0.0.1:53712)
0928 06:28:47.811554 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:47.811592 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:47.812065 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:47.812328 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:47.812332 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:47.812669 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:47.812872 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:47.812876 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:47.813570 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:47.813571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:47.814832 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:47.814834 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:47.815213 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:47.815214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:47.815344 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:47.815952 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:47.815961 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:47.817705 (+ 1744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:47.820410 (+ 2705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:47.820417 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:47.820419 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:47.820773 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:47.820989 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:47.820990 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:47.820991 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:47.821111 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:47.821468 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:47.821474 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:47.821646 (+ 172us) client_negotiation.cc:769] Sending connection context
0928 06:28:47.821757 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 06:28:47.821930 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":87,"threads_started":1}
I20260928 06:28:47.823258 30070 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.225.189" port: 35431 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:28:47.825088 30074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:47.814222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44562 (local address 127.22.225.189:35431)
0928 06:28:47.814370 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:47.814375 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:47.814519 (+ 144us) server_negotiation.cc:407] Connection header received
0928 06:28:47.814568 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:47.814570 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:47.814598 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:47.814677 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:47.815386 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:47.816380 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:47.817470 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:47.818006 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:47.818110 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:47.821063 (+ 2953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:47.821071 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:47.821071 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:47.821079 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:47.823691 (+ 2612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:47.824095 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:47.824097 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:47.824098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:47.824156 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:47.824350 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:47.824351 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:47.824351 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:47.824500 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:47.824706 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:47.824905 (+ 199us) server_negotiation.cc:299] Negotiation successful
0928 06:28:47.824987 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":91,"threads_started":1}
Sep 28 06:28:47 dist-test-slave-d59m krb5kdc[29493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576927, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.225.189@KRBTEST.COM for oryx/127.22.225.189@KRBTEST.COM
I20260928 06:28:47.831166 30075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:47.823612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44566 (local address 127.22.225.189:35431)
0928 06:28:47.823765 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:47.823767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:47.823836 (+ 69us) server_negotiation.cc:407] Connection header received
0928 06:28:47.823883 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:47.823884 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:47.823896 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:47.823974 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:47.824285 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:47.825105 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:47.826082 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:47.826286 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:47.826340 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:47.828176 (+ 1836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:47.828177 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:47.828177 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:47.828184 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:47.830074 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:47.830471 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:47.830471 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:47.830471 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:47.830524 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:47.830727 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:47.830728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:47.830728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:47.830818 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:47.830863 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:47.831021 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 06:28:47.831083 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":104,"threads_started":1}
I20260928 06:28:47.831281 30072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:47.823536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.189:35431 (local address 127.0.0.1:44566)
0928 06:28:47.823777 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:47.823788 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:47.823828 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:47.823990 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:47.823990 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:47.824131 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:47.824208 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:47.824210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:47.825201 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:47.825202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:47.826004 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:47.826007 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:47.826363 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:47.826365 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:47.826439 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:47.826788 (+ 349us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:47.826793 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:47.828083 (+ 1290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:47.830165 (+ 2082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:47.830166 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:47.830166 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:47.830391 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:47.830587 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:47.830587 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:47.830588 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:47.830656 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:47.830880 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:47.830881 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:47.830924 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 06:28:47.831008 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 06:28:47.831166 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":62}
I20260928 06:28:47.832645 30070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b: Bootstrap starting.
I20260928 06:28:47.833345 30070 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:47.833626 30070 log.cc:824] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b: Log is configured to *not* fsync() on all Append() calls
I20260928 06:28:47.834395 30070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b: No bootstrap required, opened a new log
I20260928 06:28:47.835858 30070 raft_consensus.cc:348] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c07c7b5cb55443299f41d25e3dc513d" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 41857 } } peers { permanent_uuid: "f15b6ec81d284f2baa79fa54469b144b" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 35431 } }
I20260928 06:28:47.836006 30070 raft_consensus.cc:374] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:47.836072 30070 raft_consensus.cc:729] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f15b6ec81d284f2baa79fa54469b144b, State: Initialized, Role: FOLLOWER
I20260928 06:28:47.836213 30070 consensus_queue.cc:260] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c07c7b5cb55443299f41d25e3dc513d" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 41857 } } peers { permanent_uuid: "f15b6ec81d284f2baa79fa54469b144b" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 35431 } }
I20260928 06:28:47.836467 30077 sys_catalog.cc:455] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c07c7b5cb55443299f41d25e3dc513d" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 41857 } } peers { permanent_uuid: "f15b6ec81d284f2baa79fa54469b144b" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 35431 } } }
I20260928 06:28:47.836544 30070 sys_catalog.cc:564] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:28:47.836557 30077 sys_catalog.cc:458] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b [sys.catalog]: This master's current role is: FOLLOWER
W20260928 06:28:47.838296 30088 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:28:47.838390 30088 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:28:47.839753 30002 master_runner.cc:428] Detected that this master f15b6ec81d284f2baa79fa54469b144b is joining an existing cluster
I20260928 06:28:47.839803 30002 master_runner.cc:432] Initiating AddMaster RPC to add 127.22.225.189:35431
I20260928 06:28:47.846753 30075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:47.840956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44576 (local address 127.22.225.189:35431)
0928 06:28:47.840990 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:47.840992 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:47.841099 (+ 107us) server_negotiation.cc:407] Connection header received
0928 06:28:47.841170 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:47.841171 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:47.841181 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:47.841261 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:47.841576 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:47.842033 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:47.842749 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:47.842945 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:47.843019 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:47.843937 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:47.843938 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:47.843938 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:47.843944 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:47.845652 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:47.846000 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:47.846001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:47.846001 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:47.846043 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:47.846213 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:47.846214 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:47.846214 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:47.846358 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:47.846398 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:47.846594 (+ 196us) server_negotiation.cc:299] Negotiation successful
0928 06:28:47.846666 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 06:28:47.847012 30095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:47.840790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.189:35431 (local address 127.0.0.1:44576)
0928 06:28:47.841023 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:47.841033 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:47.841113 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:47.841266 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:47.841267 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:47.841427 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:47.841500 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:47.841502 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:47.842110 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:47.842111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:47.842669 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:47.842671 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:47.843046 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:47.843047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:47.843115 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:47.843543 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:47.843548 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:47.843833 (+ 285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:47.845729 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:47.845730 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:47.845730 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:47.845931 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:47.846098 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:47.846098 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:47.846098 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:47.846149 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:47.846439 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:47.846440 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:47.846492 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 06:28:47.846851 (+ 359us) client_negotiation.cc:240] Negotiation successful
0928 06:28:47.846922 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":83,"threads_started":1}
I20260928 06:28:47.851270 29578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:47.840806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53722 (local address 127.22.225.190:41857)
0928 06:28:47.840849 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:47.840852 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:47.844724 (+ 3872us) server_negotiation.cc:407] Connection header received
0928 06:28:47.844818 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:47.844819 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:47.844833 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:47.844906 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:47.845350 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:47.846142 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:47.847356 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:47.847630 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:47.847664 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:47.848557 (+ 893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:47.848558 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:47.848559 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:47.848565 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:47.850110 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:47.850448 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:47.850449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:47.850450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:47.850514 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:47.850716 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:47.850717 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:47.850717 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:47.850828 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:47.850902 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:47.851102 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 06:28:47.851181 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:28:47.851305 30094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:47.840884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:41857 (local address 127.0.0.1:53722)
0928 06:28:47.844650 (+ 3766us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:47.844662 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:47.844752 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:47.844926 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:47.844927 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:47.845180 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:47.845251 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:47.845254 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:47.846230 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:47.846231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:47.847266 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:47.847268 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:47.847725 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:47.847725 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:47.847770 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:47.848175 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:47.848180 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:47.848445 (+ 265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:47.850194 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:47.850195 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:47.850195 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:47.850374 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:47.850592 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:47.850593 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:47.850593 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:47.850652 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:47.850892 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:47.850893 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:47.850933 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 06:28:47.851068 (+ 135us) client_negotiation.cc:240] Negotiation successful
0928 06:28:47.851195 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3749,"thread_start_us":56,"threads_started":1}
Sep 28 06:28:47 dist-test-slave-d59m krb5kdc[29493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576927, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.225.190@KRBTEST.COM for oryx/127.22.225.189@KRBTEST.COM
I20260928 06:28:47.860261 30075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:47.853349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44590 (local address 127.22.225.189:35431)
0928 06:28:47.853384 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:47.853386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:47.853569 (+ 183us) server_negotiation.cc:407] Connection header received
0928 06:28:47.853656 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:47.853657 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:47.853678 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:47.853732 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:47.854136 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:47.854614 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:47.855368 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:47.855569 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:47.855652 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:47.857660 (+ 2008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:47.857661 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:47.857662 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:47.857669 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:47.859043 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:47.859440 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:47.859441 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:47.859441 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:47.859504 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:47.859721 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:47.859721 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:47.859722 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:47.859826 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:47.859892 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:47.860082 (+ 190us) server_negotiation.cc:299] Negotiation successful
0928 06:28:47.860171 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:28:47.860270 30096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:47.853322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.189:35431 (local address 127.0.0.1:44590)
0928 06:28:47.853450 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:47.853500 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:47.853592 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:47.853738 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:47.853742 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:47.854006 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:47.854048 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:47.854049 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:47.854695 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:47.854696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:47.855284 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:47.855286 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:47.855656 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:47.855657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:47.855759 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:47.856215 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:47.856220 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:47.857567 (+ 1347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:47.859144 (+ 1577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:47.859152 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:47.859154 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:47.859362 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:47.859585 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:47.859586 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:47.859587 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:47.859651 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:47.859897 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:47.859905 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:47.859955 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 06:28:47.860055 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 06:28:47.860176 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":83,"threads_started":1}
I20260928 06:28:47.860881 29538 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.22.225.189:35431: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "f15b6ec81d284f2baa79fa54469b144b"
member_type: NON_VOTER
last_known_addr {
host: "127.22.225.189"
port: 35431
}
attrs {
promote: true
}
}
dest_uuid: "0c07c7b5cb55443299f41d25e3dc513d"
cas_config_opid_index: -1
I20260928 06:28:47.861215 29538 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c07c7b5cb55443299f41d25e3dc513d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0c07c7b5cb55443299f41d25e3dc513d" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 41857 } } peers { permanent_uuid: "f15b6ec81d284f2baa79fa54469b144b" member_type: NON_VOTER last_known_addr { host: "127.22.225.189" port: 35431 } attrs { promote: true } }
I20260928 06:28:47.861675 30098 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0c07c7b5cb55443299f41d25e3dc513d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER f15b6ec81d284f2baa79fa54469b144b (127.22.225.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0c07c7b5cb55443299f41d25e3dc513d" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 41857 } } peers { permanent_uuid: "f15b6ec81d284f2baa79fa54469b144b" member_type: NON_VOTER last_known_addr { host: "127.22.225.189" port: 35431 } attrs { promote: true } } }
I20260928 06:28:47.862023 30044 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:47.862919 30044 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b [term 1 FOLLOWER]: Refusing update from remote peer 0c07c7b5cb55443299f41d25e3dc513d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 06:28:47.863046 30098 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.22.225.189:35431
I20260928 06:28:47.863312 30002 master.cc:537] Master@127.22.225.189:35431 shutting down...
I20260928 06:28:47.863324 30097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c07c7b5cb55443299f41d25e3dc513d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0c07c7b5cb55443299f41d25e3dc513d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0c07c7b5cb55443299f41d25e3dc513d" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 41857 } } peers { permanent_uuid: "f15b6ec81d284f2baa79fa54469b144b" member_type: NON_VOTER last_known_addr { host: "127.22.225.189" port: 35431 } attrs { promote: true } } }
I20260928 06:28:47.863445 30097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c07c7b5cb55443299f41d25e3dc513d [sys.catalog]: This master's current role is: LEADER
I20260928 06:28:47.863315 30098 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0c07c7b5cb55443299f41d25e3dc513d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f15b6ec81d284f2baa79fa54469b144b" member_type: NON_VOTER last_known_addr { host: "127.22.225.189" port: 35431 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 06:28:47.864759 29524 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260928 06:28:47.866266 29524 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0c07c7b5cb55443299f41d25e3dc513d -> Peer f15b6ec81d284f2baa79fa54469b144b (127.22.225.189:35431): Couldn't send request to peer f15b6ec81d284f2baa79fa54469b144b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 06:28:47.866711 30002 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:28:47.866813 30002 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:28:47.866866 30002 tablet_replica.cc:330] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b: stopping tablet replica
I20260928 06:28:47.879354 30002 master.cc:552] Master@127.22.225.189:35431 shutdown complete.
I20260928 06:28:47.879427 30002 master_runner.cc:305] Clearing existing system tablet
I20260928 06:28:47.879566 30002 fs_manager.cc:359] Metadata directory not provided
I20260928 06:28:47.879627 30002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:28:47.880250 30002 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:28:47.880754 30103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:47.880908 30002 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:28:47.881042 30002 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:28:47.881088 30002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/master-1/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/master-1/wal
uuid: "f15b6ec81d284f2baa79fa54469b144b"
format_stamp: "Formatted at 2026-09-28 06:28:47 on dist-test-slave-d59m"
server_key: "8d65bb6f3c2eef57dceba1167d7cb667"
server_key_iv: "106153f3750ee2bc0e0460965f2185c2"
server_key_version: "encryptionkey@0"
I20260928 06:28:47.882088 30002 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:28:47.883649 30002 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 06:28:47.883737 30002 log.cc:1196] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b: Deleting WAL directory at /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 06:28:47.883934 30002 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b: Deleting consensus metadata
I20260928 06:28:47.884086 30002 master_runner.cc:315] Copying system tablet from 127.22.225.190:41857
I20260928 06:28:47.884799 30002 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b: tablet copy: Beginning tablet copy session from remote peer at address 127.22.225.190:41857
I20260928 06:28:47.891342 29578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:47.885191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53728 (local address 127.22.225.190:41857)
0928 06:28:47.885243 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:47.885245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:47.885373 (+ 128us) server_negotiation.cc:407] Connection header received
0928 06:28:47.885445 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:47.885446 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:47.885457 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:47.885515 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:47.885862 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:47.886461 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:47.887251 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:47.887524 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:47.887624 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:47.888644 (+ 1020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:47.888646 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:47.888646 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:47.888653 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:47.890174 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:47.890547 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:47.890548 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:47.890549 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:47.890607 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:47.890811 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:47.890812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:47.890813 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:47.890927 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:47.891018 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:47.891173 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 06:28:47.891250 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 06:28:47.891358 30109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:47.885140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:41857 (local address 127.0.0.1:53728)
0928 06:28:47.885290 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:47.885308 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:47.885388 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:47.885562 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:47.885563 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:47.885724 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:47.885772 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:47.885775 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:47.886546 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:47.886547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:47.887166 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:47.887168 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:47.887610 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:47.887612 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:47.887679 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:47.888193 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:47.888199 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:47.888516 (+ 317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:47.890276 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:47.890278 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:47.890278 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:47.890454 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:47.890680 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:47.890681 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:47.890681 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:47.890739 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:47.891010 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:47.891011 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:47.891057 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 06:28:47.891147 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 06:28:47.891250 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":66,"threads_started":1}
I20260928 06:28:47.892042 29558 tablet_copy_service.cc:140] P 0c07c7b5cb55443299f41d25e3dc513d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer f15b6ec81d284f2baa79fa54469b144b ({username='oryx', principal='oryx/127.22.225.189@KRBTEST.COM'} at 127.0.0.1:53728)
I20260928 06:28:47.892160 29558 tablet_copy_service.cc:161] P 0c07c7b5cb55443299f41d25e3dc513d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer f15b6ec81d284f2baa79fa54469b144b at {username='oryx', principal='oryx/127.22.225.189@KRBTEST.COM'} at 127.0.0.1:53728: session id = f15b6ec81d284f2baa79fa54469b144b-00000000000000000000000000000000
I20260928 06:28:47.893098 29558 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 0c07c7b5cb55443299f41d25e3dc513d: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 06:28:47.893810 30002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 06:28:47.895210 30002 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b: tablet copy: Starting download of 0 data blocks...
I20260928 06:28:47.895370 30002 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b: tablet copy: Starting download of 1 WAL segments...
I20260928 06:28:47.896242 30002 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 06:28:47.897174 29558 tablet_copy_service.cc:342] P 0c07c7b5cb55443299f41d25e3dc513d: Request end of tablet copy session f15b6ec81d284f2baa79fa54469b144b-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.22.225.189@KRBTEST.COM'} at 127.0.0.1:53728
I20260928 06:28:47.897236 29558 tablet_copy_service.cc:433] P 0c07c7b5cb55443299f41d25e3dc513d: ending tablet copy session f15b6ec81d284f2baa79fa54469b144b-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer f15b6ec81d284f2baa79fa54469b144b
I20260928 06:28:47.900292 30002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:28:47.901430 30111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:47.901527 30110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:47.901834 30113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:47.904796 30002 server_base.cc:1034] running on GCE node
I20260928 06:28:47.904906 30002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:28:47.905059 30002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:28:47.906198 30002 hybrid_clock.cc:648] HybridClock initialized: now 1790576927906173 us; error 40 us; skew 500 ppm
Sep 28 06:28:47 dist-test-slave-d59m krb5kdc[29493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576927, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.225.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:28:47.907711 30002 init.cc:377] Logged in from keytab as oryx/127.22.225.189@KRBTEST.COM (short username oryx)
I20260928 06:28:47.907963 30002 webserver.cc:460] Webserver started at http://127.22.225.189:34119/ using document root <none> and password file <none>
I20260928 06:28:47.908090 30002 fs_manager.cc:359] Metadata directory not provided
I20260928 06:28:47.908126 30002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:28:47.908905 30002 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:28:47.909534 30120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:47.909710 30002 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:47.909765 30002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/master-1/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/master-1/wal
uuid: "f15b6ec81d284f2baa79fa54469b144b"
format_stamp: "Formatted at 2026-09-28 06:28:47 on dist-test-slave-d59m"
server_key: "8d65bb6f3c2eef57dceba1167d7cb667"
server_key_iv: "106153f3750ee2bc0e0460965f2185c2"
server_key_version: "encryptionkey@0"
I20260928 06:28:47.909821 30002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:28:47.927918 30002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:28:47.929888 30002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:28:47.933151 30002 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.189:35431
I20260928 06:28:47.933449 30172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.189:35431 every 8 connection(s)
I20260928 06:28:47.933557 30002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/master-1/data/info.pb
I20260928 06:28:47.934492 30173 sys_catalog.cc:263] Verifying existing consensus state
I20260928 06:28:47.935003 30173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b: Bootstrap starting.
I20260928 06:28:47.937955 30173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:28:47.938206 30173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b: Bootstrap complete.
I20260928 06:28:47.938402 30173 raft_consensus.cc:348] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0c07c7b5cb55443299f41d25e3dc513d" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 41857 } } peers { permanent_uuid: "f15b6ec81d284f2baa79fa54469b144b" member_type: NON_VOTER last_known_addr { host: "127.22.225.189" port: 35431 } attrs { promote: true } }
I20260928 06:28:47.938496 30173 raft_consensus.cc:729] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f15b6ec81d284f2baa79fa54469b144b, State: Initialized, Role: LEARNER
I20260928 06:28:47.938531 30173 consensus_queue.cc:260] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0c07c7b5cb55443299f41d25e3dc513d" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 41857 } } peers { permanent_uuid: "f15b6ec81d284f2baa79fa54469b144b" member_type: NON_VOTER last_known_addr { host: "127.22.225.189" port: 35431 } attrs { promote: true } }
I20260928 06:28:47.938701 30173 sys_catalog.cc:564] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:28:47.938714 30176 sys_catalog.cc:455] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0c07c7b5cb55443299f41d25e3dc513d" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 41857 } } peers { permanent_uuid: "f15b6ec81d284f2baa79fa54469b144b" member_type: NON_VOTER last_known_addr { host: "127.22.225.189" port: 35431 } attrs { promote: true } } }
I20260928 06:28:47.938954 30176 sys_catalog.cc:458] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b [sys.catalog]: This master's current role is: LEARNER
I20260928 06:28:47.940600 30187 catalog_manager.cc:1259] Loaded cluster ID: 3fb00626f39f44199dbf917853eb1818
I20260928 06:28:47.940654 30187 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b: loading cluster ID for follower catalog manager: success
I20260928 06:28:47.941699 30187 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b: acquiring CA information for follower catalog manager: success
I20260928 06:28:47.942004 30187 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 06:28:47.989367 30188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:47.977195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44602 (local address 127.22.225.189:35431)
0928 06:28:47.977341 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:47.977343 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:47.977352 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:28:47.977379 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:47.977379 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:47.977391 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:47.977456 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:47.978154 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:47.979598 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:47.980740 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:47.981042 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:47.981122 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:47.985917 (+ 4795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:47.985919 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:47.985920 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:47.985929 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:47.987914 (+ 1985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:47.988361 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:47.988363 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:47.988363 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:47.988485 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:47.988755 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:47.988756 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:47.988756 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:47.988888 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:47.988997 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:47.989161 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 06:28:47.989248 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":89,"threads_started":1}
WARNING: no policy specified for oryx/127.22.225.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.225.188@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.225.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/krb5kdc/oryx_127.22.225.188.keytab.
Entry for principal oryx/127.22.225.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/krb5kdc/oryx_127.22.225.188.keytab.
I20260928 06:28:48.019670 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.188:34529
--webserver_interface=127.22.225.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/krb5kdc/oryx_127.22.225.188.keytab
--principal=oryx/127.22.225.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:36117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.225.190:41857,127.22.225.189:35431,127.22.225.188:34529
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:28:48.108431 30191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:28:48.108794 30191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:28:48.108847 30191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:28:48.110198 30191 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:28:48.110258 30191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:28:48.110283 30191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:28:48.110307 30191 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:28:48.110333 30191 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:28:48.111939 30191 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:36117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.22.225.190:41857,127.22.225.189:35431,127.22.225.188:34529
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/krb5kdc/oryx_127.22.225.188.keytab
--principal=oryx/127.22.225.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.188:34529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.22.225.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:28:48.112222 30191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:28:48.112450 30191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:28:48.117271 30196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:48.117394 30197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:48.117278 30199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:48.117771 30191 server_base.cc:1034] running on GCE node
I20260928 06:28:48.118013 30191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:28:48.118310 30191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:28:48.119482 30191 hybrid_clock.cc:648] HybridClock initialized: now 1790576928119464 us; error 39 us; skew 500 ppm
Sep 28 06:28:48 dist-test-slave-d59m krb5kdc[29493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576928, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.225.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:28:48.122207 30191 init.cc:377] Logged in from keytab as oryx/127.22.225.188@KRBTEST.COM (short username oryx)
I20260928 06:28:48.122674 30191 webserver.cc:460] Webserver started at http://127.22.225.188:40173/ using document root <none> and password file <none>
I20260928 06:28:48.122938 30191 fs_manager.cc:359] Metadata directory not provided
I20260928 06:28:48.123014 30191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:28:48.123167 30191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:28:48.124254 30191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/master-2/data/instance:
uuid: "7b496d70bc7c49da9032416864dc8ff5"
format_stamp: "Formatted at 2026-09-28 06:28:48 on dist-test-slave-d59m"
server_key: "9fc870c68ceb2f0681391b3c99dd4c69"
server_key_iv: "358f4d54f4175617b22a59f90d7e60d1"
server_key_version: "encryptionkey@0"
I20260928 06:28:48.124711 30191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/master-2/wal/instance:
uuid: "7b496d70bc7c49da9032416864dc8ff5"
format_stamp: "Formatted at 2026-09-28 06:28:48 on dist-test-slave-d59m"
server_key: "9fc870c68ceb2f0681391b3c99dd4c69"
server_key_iv: "358f4d54f4175617b22a59f90d7e60d1"
server_key_version: "encryptionkey@0"
I20260928 06:28:48.126127 30191 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:28:48.126940 30206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:48.127118 30191 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:28:48.127216 30191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/master-2/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/master-2/wal
uuid: "7b496d70bc7c49da9032416864dc8ff5"
format_stamp: "Formatted at 2026-09-28 06:28:48 on dist-test-slave-d59m"
server_key: "9fc870c68ceb2f0681391b3c99dd4c69"
server_key_iv: "358f4d54f4175617b22a59f90d7e60d1"
server_key_version: "encryptionkey@0"
I20260928 06:28:48.127301 30191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:28:48.140820 30191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:28:48.143319 30191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:28:48.143498 30191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:28:48.147920 30191 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.188:34529
I20260928 06:28:48.147976 30258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.188:34529 every 8 connection(s)
I20260928 06:28:48.148329 30191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/master-2/data/info.pb
I20260928 06:28:48.149612 30259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:48.151916 30259 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.225.190" port: 41857 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:28:48.154811 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 30191
Sep 28 06:28:48 dist-test-slave-d59m krb5kdc[29493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576928, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.225.188@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM
Sep 28 06:28:48 dist-test-slave-d59m krb5kdc[29493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.225.188@KRBTEST.COM
I20260928 06:28:48.162132 29578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:48.152846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53730 (local address 127.22.225.190:41857)
0928 06:28:48.152921 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:48.152924 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:48.153131 (+ 207us) server_negotiation.cc:407] Connection header received
0928 06:28:48.153765 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:48.153766 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:48.153775 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:48.153870 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:48.154696 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.155393 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.156525 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.156777 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.156861 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:48.158990 (+ 2129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:48.158992 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:48.158992 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:48.159000 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:48.160699 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:48.161108 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:48.161109 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:48.161110 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:48.161172 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:48.161432 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:48.161433 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:48.161434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:48.161552 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:48.161628 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:48.161954 (+ 326us) server_negotiation.cc:299] Negotiation successful
0928 06:28:48.162039 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 06:28:48.162326 30261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:48.152801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:41857 (local address 127.0.0.1:53730)
0928 06:28:48.152981 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:48.153037 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:48.153660 (+ 623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:48.153946 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:48.153950 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:48.154303 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:48.154578 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:48.154584 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.155516 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.155517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:48.156413 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:48.156415 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.156888 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.156889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:48.157004 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:48.157546 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:48.157554 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:48.158895 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:48.160792 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:48.160799 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:48.160801 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:48.161028 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:48.161238 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:48.161239 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:48.161240 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:48.161353 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:48.161618 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:48.161623 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:48.161796 (+ 173us) client_negotiation.cc:769] Sending connection context
0928 06:28:48.161928 (+ 132us) client_negotiation.cc:240] Negotiation successful
0928 06:28:48.162085 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":89,"threads_started":1}
I20260928 06:28:48.162884 30259 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.225.189" port: 35431 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:28:48.166354 30262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:48.155494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42174 (local address 127.22.225.188:34529)
0928 06:28:48.155676 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:48.155680 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:48.155697 (+ 17us) server_negotiation.cc:407] Connection header received
0928 06:28:48.155717 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:48.155719 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:48.155737 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:48.155815 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:48.156559 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.157726 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.158813 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.159279 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.159358 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:48.162317 (+ 2959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:48.162324 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:48.162324 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:48.162331 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:48.164740 (+ 2409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:48.165124 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:48.165127 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:48.165128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:48.165176 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:48.165384 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:48.165385 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:48.165385 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:48.165558 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:48.166072 (+ 514us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:48.166120 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 06:28:48.166257 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":99,"threads_started":1}
Sep 28 06:28:48 dist-test-slave-d59m krb5kdc[29493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576928, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.225.188@KRBTEST.COM for oryx/127.22.225.189@KRBTEST.COM
I20260928 06:28:48.170032 30261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:48.163156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.189:35431 (local address 127.0.0.1:44616)
0928 06:28:48.163198 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:48.163207 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:48.163238 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:48.163380 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:48.163381 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:48.163512 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:48.163562 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:48.163563 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.164311 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.164312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:48.164973 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:48.164984 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.165570 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.165570 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:48.165646 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:48.166053 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:48.166058 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:48.167324 (+ 1266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:48.169145 (+ 1821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:48.169145 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:48.169146 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:48.169297 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:48.169505 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:48.169505 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:48.169506 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:48.169552 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:48.169825 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:48.169826 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:48.169857 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 06:28:48.169902 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 06:28:48.169964 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 06:28:48.170146 30188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:48.163225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44616 (local address 127.22.225.189:35431)
0928 06:28:48.163277 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:48.163279 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:48.163286 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:28:48.163309 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:48.163310 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:48.163319 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:48.163373 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:48.163653 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.164210 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.165081 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.165409 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.165484 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:48.167411 (+ 1927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:48.167413 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:48.167413 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:48.167424 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:48.169054 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:48.169369 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:48.169370 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:48.169370 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:48.169433 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:48.169615 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:48.169616 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:48.169617 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:48.169755 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:48.169839 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:48.169938 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 06:28:48.170041 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 06:28:48.170521 30259 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.225.188" port: 34529 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:28:48 dist-test-slave-d59m krb5kdc[29493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576928, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.225.188@KRBTEST.COM for oryx/127.22.225.188@KRBTEST.COM
I20260928 06:28:48.177129 30261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:48.170787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.188:34529 (local address 127.0.0.1:42176)
0928 06:28:48.170846 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:48.170859 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:48.170915 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:48.171037 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:48.171039 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:48.171189 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:48.171223 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:48.171225 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.171840 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.171841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:48.172671 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:48.172674 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.173004 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.173005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:48.173074 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:48.173418 (+ 344us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:48.173423 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:48.174617 (+ 1194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:48.176134 (+ 1517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:48.176135 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:48.176135 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:48.176292 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:48.176494 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:48.176494 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:48.176495 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:48.176547 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:48.176824 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:48.176824 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:48.176858 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 06:28:48.176927 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 06:28:48.177016 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 06:28:48.177173 30262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:48.170817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42176 (local address 127.22.225.188:34529)
0928 06:28:48.170847 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:48.170849 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:48.170899 (+ 50us) server_negotiation.cc:407] Connection header received
0928 06:28:48.170969 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:48.170970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:48.170978 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:48.171023 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:48.171298 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.171758 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.172744 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.172931 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.172984 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:48.174701 (+ 1717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:48.174702 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:48.174703 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:48.174710 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:48.176039 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:48.176374 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:48.176375 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:48.176376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:48.176427 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:48.176658 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:48.176659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:48.176659 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:48.176758 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:48.176833 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:48.176948 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 06:28:48.177063 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":13}
I20260928 06:28:48.178485 30259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b496d70bc7c49da9032416864dc8ff5: Bootstrap starting.
I20260928 06:28:48.179297 30259 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b496d70bc7c49da9032416864dc8ff5: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:48.179592 30259 log.cc:824] T 00000000000000000000000000000000 P 7b496d70bc7c49da9032416864dc8ff5: Log is configured to *not* fsync() on all Append() calls
I20260928 06:28:48.180401 30259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b496d70bc7c49da9032416864dc8ff5: No bootstrap required, opened a new log
I20260928 06:28:48.181870 30259 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b496d70bc7c49da9032416864dc8ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c07c7b5cb55443299f41d25e3dc513d" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 41857 } } peers { permanent_uuid: "f15b6ec81d284f2baa79fa54469b144b" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 35431 } } peers { permanent_uuid: "7b496d70bc7c49da9032416864dc8ff5" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 34529 } }
I20260928 06:28:48.182014 30259 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b496d70bc7c49da9032416864dc8ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:48.182062 30259 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b496d70bc7c49da9032416864dc8ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b496d70bc7c49da9032416864dc8ff5, State: Initialized, Role: FOLLOWER
I20260928 06:28:48.182205 30259 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b496d70bc7c49da9032416864dc8ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c07c7b5cb55443299f41d25e3dc513d" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 41857 } } peers { permanent_uuid: "f15b6ec81d284f2baa79fa54469b144b" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 35431 } } peers { permanent_uuid: "7b496d70bc7c49da9032416864dc8ff5" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 34529 } }
I20260928 06:28:48.182482 30264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b496d70bc7c49da9032416864dc8ff5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c07c7b5cb55443299f41d25e3dc513d" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 41857 } } peers { permanent_uuid: "f15b6ec81d284f2baa79fa54469b144b" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 35431 } } peers { permanent_uuid: "7b496d70bc7c49da9032416864dc8ff5" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 34529 } } }
I20260928 06:28:48.182548 30259 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b496d70bc7c49da9032416864dc8ff5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:28:48.182569 30264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b496d70bc7c49da9032416864dc8ff5 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 06:28:48.184396 30275 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7b496d70bc7c49da9032416864dc8ff5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:28:48.184837 30275 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:28:48.186896 30191 master_runner.cc:428] Detected that this master 7b496d70bc7c49da9032416864dc8ff5 is joining an existing cluster
I20260928 06:28:48.187016 30191 master_runner.cc:432] Initiating AddMaster RPC to add 127.22.225.188:34529
I20260928 06:28:48.195547 30281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:48.188164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.189:35431 (local address 127.0.0.1:44624)
0928 06:28:48.188361 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:48.188372 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:48.188428 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:48.188624 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:48.188625 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:48.188824 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:48.188898 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:48.188900 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.189865 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.189866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:48.190785 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:48.190787 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.191295 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.191296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:48.191373 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:48.191872 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:48.191877 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:48.192272 (+ 395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:48.194353 (+ 2081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:48.194354 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:48.194355 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:48.194577 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:48.194806 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:48.194807 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:48.194807 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:48.194873 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:48.195172 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:48.195174 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:48.195223 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 06:28:48.195323 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 06:28:48.195422 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":84,"threads_started":1}
I20260928 06:28:48.195887 30188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:48.188223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44624 (local address 127.22.225.189:35431)
0928 06:28:48.188263 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:48.188264 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:48.188417 (+ 153us) server_negotiation.cc:407] Connection header received
0928 06:28:48.188480 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:48.188481 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:48.188492 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:48.188565 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:48.188985 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.189770 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.190882 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.191198 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.191275 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:48.192371 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:48.192372 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:48.192373 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:48.192380 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:48.194260 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:48.194662 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:48.194664 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:48.194664 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:48.194729 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:48.194946 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:48.194947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:48.194947 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:48.195095 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:48.195168 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:48.195715 (+ 547us) server_negotiation.cc:299] Negotiation successful
0928 06:28:48.195797 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:28:48.202979 30262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:48.193152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42182 (local address 127.22.225.188:34529)
0928 06:28:48.193194 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:48.193196 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:48.196777 (+ 3581us) server_negotiation.cc:407] Connection header received
0928 06:28:48.196864 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:48.196865 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:48.196873 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:48.196907 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:48.197355 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.198056 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.199083 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.199375 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.199413 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:48.200471 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:48.200472 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:48.200472 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:48.200478 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:48.201896 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:48.202208 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:48.202209 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:48.202209 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:48.202257 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:48.202469 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:48.202470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:48.202471 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:48.202579 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:48.202654 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:48.202777 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 06:28:48.202896 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":25}
I20260928 06:28:48.202978 30282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:48.193137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.188:34529 (local address 127.0.0.1:42182)
0928 06:28:48.196696 (+ 3559us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:48.196707 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:48.196800 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:48.196958 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:48.196959 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:48.197155 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:48.197229 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:48.197231 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.198148 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.198148 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:48.198973 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:48.198976 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.199481 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.199481 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:48.199542 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:48.199944 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:48.199949 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:48.200372 (+ 423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:48.201959 (+ 1587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:48.201960 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:48.201961 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:48.202121 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:48.202344 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:48.202345 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:48.202345 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:48.202398 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:48.202645 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:48.202646 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:48.202682 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 06:28:48.202749 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 06:28:48.202848 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3534,"thread_start_us":103,"threads_started":1}
I20260928 06:28:48.217805 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 29600
I20260928 06:28:48.223938 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.129:40749
--local_ip_for_outbound_sockets=127.22.225.129
--tserver_master_addrs=127.22.225.190:41857
--webserver_port=40407
--webserver_interface=127.22.225.129
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/krb5kdc/oryx_127.22.225.129.keytab
--principal=oryx/127.22.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:36117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.22.225.190:41857,127.22.225.189:35431
--tserver_master_addrs=127.22.225.190:41857,127.22.225.189:35431,127.22.225.188:34529 with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:28:48.232908 29578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:48.192819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53746 (local address 127.22.225.190:41857)
0928 06:28:48.192864 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:48.192866 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:48.196281 (+ 3415us) server_negotiation.cc:407] Connection header received
0928 06:28:48.196379 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:48.196380 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:48.196391 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:48.196421 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:48.196813 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.197582 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.198611 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.198916 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.198955 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:48.229143 (+ 30188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:48.229147 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:48.229147 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:48.229158 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:48.231377 (+ 2219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:48.231874 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:48.231876 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:48.231876 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:48.231940 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:48.232190 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:48.232191 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:48.232191 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:48.232355 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:48.232406 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:48.232670 (+ 264us) server_negotiation.cc:299] Negotiation successful
0928 06:28:48.232766 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:28:48.233249 30281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:48.192933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:41857 (local address 127.0.0.1:53746)
0928 06:28:48.196231 (+ 3298us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:48.196241 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:48.196307 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:48.196480 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:48.196481 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:48.196669 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:48.196718 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:48.196720 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.197680 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.197681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:48.198514 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:48.198516 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.199040 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.199041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:48.199103 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:48.199520 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:48.199525 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:48.228971 (+ 29446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:48.231518 (+ 2547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:48.231519 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:48.231520 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:48.231774 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:48.232023 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:48.232024 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:48.232024 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:48.232107 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:48.232461 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:48.232462 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:48.232516 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 06:28:48.233019 (+ 503us) client_negotiation.cc:240] Negotiation successful
0928 06:28:48.233147 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3288,"thread_start_us":77,"threads_started":1}
Sep 28 06:28:48 dist-test-slave-d59m krb5kdc[29493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576927, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.225.190@KRBTEST.COM for oryx/127.22.225.188@KRBTEST.COM
I20260928 06:28:48.243794 30096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:48.235519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.188:34529 (local address 127.0.0.1:42192)
0928 06:28:48.235564 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:48.235576 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:48.235623 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:48.235920 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:48.235921 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:48.236092 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:48.236138 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:48.236139 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.237092 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.237093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:48.237917 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:48.237919 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.238465 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.238466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:48.238536 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:48.238994 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:48.239000 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:48.240698 (+ 1698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:48.242505 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:48.242506 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:48.242506 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:48.242730 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:48.242954 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:48.242955 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:48.242956 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:48.243020 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:48.243323 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:48.243323 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:48.243366 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 06:28:48.243425 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 06:28:48.243642 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 06:28:48.244303 30262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:48.235701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42192 (local address 127.22.225.188:34529)
0928 06:28:48.235764 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:48.235766 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:48.235774 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:28:48.235805 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:48.235806 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:48.235816 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:48.235869 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:48.236215 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.236978 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.238068 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.238361 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.238416 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:48.240794 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:48.240796 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:48.240796 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:48.240803 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:48.242413 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:48.242821 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:48.242822 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:48.242822 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:48.242879 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:48.243112 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:48.243113 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:48.243113 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:48.243240 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:48.243297 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:48.244067 (+ 770us) server_negotiation.cc:299] Negotiation successful
0928 06:28:48.244166 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 06:28:48.244906 29538 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.22.225.188:34529: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "7b496d70bc7c49da9032416864dc8ff5"
member_type: NON_VOTER
last_known_addr {
host: "127.22.225.188"
port: 34529
}
attrs {
promote: true
}
}
dest_uuid: "0c07c7b5cb55443299f41d25e3dc513d"
cas_config_opid_index: 5
I20260928 06:28:48.245187 29538 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c07c7b5cb55443299f41d25e3dc513d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0c07c7b5cb55443299f41d25e3dc513d" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 41857 } } peers { permanent_uuid: "f15b6ec81d284f2baa79fa54469b144b" member_type: NON_VOTER last_known_addr { host: "127.22.225.189" port: 35431 } attrs { promote: true } } peers { permanent_uuid: "7b496d70bc7c49da9032416864dc8ff5" member_type: NON_VOTER last_known_addr { host: "127.22.225.188" port: 34529 } attrs { promote: true } }
I20260928 06:28:48.245544 30098 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0c07c7b5cb55443299f41d25e3dc513d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 7b496d70bc7c49da9032416864dc8ff5 (127.22.225.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0c07c7b5cb55443299f41d25e3dc513d" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 41857 } } peers { permanent_uuid: "f15b6ec81d284f2baa79fa54469b144b" member_type: NON_VOTER last_known_addr { host: "127.22.225.189" port: 35431 } attrs { promote: true } } peers { permanent_uuid: "7b496d70bc7c49da9032416864dc8ff5" member_type: NON_VOTER last_known_addr { host: "127.22.225.188" port: 34529 } attrs { promote: true } } }
I20260928 06:28:48.245864 30233 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b496d70bc7c49da9032416864dc8ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:48.246613 30098 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.22.225.188:34529
I20260928 06:28:48.246703 30098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c07c7b5cb55443299f41d25e3dc513d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0c07c7b5cb55443299f41d25e3dc513d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0c07c7b5cb55443299f41d25e3dc513d" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 41857 } } peers { permanent_uuid: "f15b6ec81d284f2baa79fa54469b144b" member_type: NON_VOTER last_known_addr { host: "127.22.225.189" port: 35431 } attrs { promote: true } } peers { permanent_uuid: "7b496d70bc7c49da9032416864dc8ff5" member_type: NON_VOTER last_known_addr { host: "127.22.225.188" port: 34529 } attrs { promote: true } } }
I20260928 06:28:48.246780 30098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c07c7b5cb55443299f41d25e3dc513d [sys.catalog]: This master's current role is: LEADER
I20260928 06:28:48.246809 30233 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7b496d70bc7c49da9032416864dc8ff5 [term 1 FOLLOWER]: Refusing update from remote peer 0c07c7b5cb55443299f41d25e3dc513d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 06:28:48.247080 30099 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0c07c7b5cb55443299f41d25e3dc513d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b496d70bc7c49da9032416864dc8ff5" member_type: NON_VOTER last_known_addr { host: "127.22.225.188" port: 34529 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:28:48.247174 30191 master.cc:537] Master@127.22.225.188:34529 shutting down...
W20260928 06:28:48.247889 29525 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0c07c7b5cb55443299f41d25e3dc513d -> Peer 7b496d70bc7c49da9032416864dc8ff5 (127.22.225.188:34529): Couldn't send request to peer 7b496d70bc7c49da9032416864dc8ff5. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 06:28:48.250512 30191 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 7b496d70bc7c49da9032416864dc8ff5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:28:48.250599 30191 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 7b496d70bc7c49da9032416864dc8ff5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:28:48.250643 30191 tablet_replica.cc:330] T 00000000000000000000000000000000 P 7b496d70bc7c49da9032416864dc8ff5: stopping tablet replica
I20260928 06:28:48.254870 30096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:48.246984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.189:35431 (local address 127.0.0.1:44630)
0928 06:28:48.249944 (+ 2960us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:48.249957 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:48.250011 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:48.251432 (+ 1421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:48.251434 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:48.251436 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:28:48.251483 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:48.251485 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.252403 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.252404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:48.253786 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:48.253789 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.254626 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.254627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:48.254687 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:48.254710 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 06:28:48.254758 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 06:28:48.254798 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2939}
I20260928 06:28:48.255195 30188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:48.247053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44630 (local address 127.22.225.189:35431)
0928 06:28:48.247101 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:48.247104 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:48.250080 (+ 2976us) server_negotiation.cc:407] Connection header received
0928 06:28:48.250111 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:48.250112 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:48.250136 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:48.250196 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:28:48.251574 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.252309 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.253888 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.254459 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.254528 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:48.254591 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:48.255099 (+ 508us) server_negotiation.cc:299] Negotiation successful
0928 06:28:48.255136 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 06:28:48.256331 30147 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b [term 1 LEARNER]: Refusing update from remote peer 0c07c7b5cb55443299f41d25e3dc513d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 06:28:48.256546 30099 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0c07c7b5cb55443299f41d25e3dc513d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f15b6ec81d284f2baa79fa54469b144b" member_type: NON_VOTER last_known_addr { host: "127.22.225.189" port: 35431 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260928 06:28:48.257144 30147 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 7b496d70bc7c49da9032416864dc8ff5 (127.22.225.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0c07c7b5cb55443299f41d25e3dc513d" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 41857 } } peers { permanent_uuid: "f15b6ec81d284f2baa79fa54469b144b" member_type: NON_VOTER last_known_addr { host: "127.22.225.189" port: 35431 } attrs { promote: true } } peers { permanent_uuid: "7b496d70bc7c49da9032416864dc8ff5" member_type: NON_VOTER last_known_addr { host: "127.22.225.188" port: 34529 } attrs { promote: true } } }
I20260928 06:28:48.258343 30286 sys_catalog.cc:455] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0c07c7b5cb55443299f41d25e3dc513d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0c07c7b5cb55443299f41d25e3dc513d" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 41857 } } peers { permanent_uuid: "f15b6ec81d284f2baa79fa54469b144b" member_type: NON_VOTER last_known_addr { host: "127.22.225.189" port: 35431 } attrs { promote: true } } peers { permanent_uuid: "7b496d70bc7c49da9032416864dc8ff5" member_type: NON_VOTER last_known_addr { host: "127.22.225.188" port: 34529 } attrs { promote: true } } }
I20260928 06:28:48.258471 30286 sys_catalog.cc:458] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b [sys.catalog]: This master's current role is: LEARNER
I20260928 06:28:48.258478 30176 sys_catalog.cc:455] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c07c7b5cb55443299f41d25e3dc513d. Latest consensus state: current_term: 1 leader_uuid: "0c07c7b5cb55443299f41d25e3dc513d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0c07c7b5cb55443299f41d25e3dc513d" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 41857 } } peers { permanent_uuid: "f15b6ec81d284f2baa79fa54469b144b" member_type: NON_VOTER last_known_addr { host: "127.22.225.189" port: 35431 } attrs { promote: true } } peers { permanent_uuid: "7b496d70bc7c49da9032416864dc8ff5" member_type: NON_VOTER last_known_addr { host: "127.22.225.188" port: 34529 } attrs { promote: true } } }
I20260928 06:28:48.258616 30176 sys_catalog.cc:458] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b [sys.catalog]: This master's current role is: LEARNER
I20260928 06:28:48.259068 30099 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 0c07c7b5cb55443299f41d25e3dc513d: attempting to promote NON_VOTER f15b6ec81d284f2baa79fa54469b144b to VOTER
I20260928 06:28:48.259261 30099 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c07c7b5cb55443299f41d25e3dc513d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0c07c7b5cb55443299f41d25e3dc513d" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 41857 } } peers { permanent_uuid: "f15b6ec81d284f2baa79fa54469b144b" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 35431 } attrs { promote: false } } peers { permanent_uuid: "7b496d70bc7c49da9032416864dc8ff5" member_type: NON_VOTER last_known_addr { host: "127.22.225.188" port: 34529 } attrs { promote: true } }
I20260928 06:28:48.259691 30096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:48.259578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.188:34529 (local address 127.0.0.1:42198)
0928 06:28:48.259621 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:48.259662 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.188:34529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 06:28:48.259814 30147 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b [term 1 LEARNER]: Refusing update from remote peer 0c07c7b5cb55443299f41d25e3dc513d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260928 06:28:48.259948 30096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:48.259886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.188:34529 (local address 127.0.0.1:42206)
0928 06:28:48.259912 (+ 26us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:48.259930 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.188:34529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260928 06:28:48.260016 29525 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0c07c7b5cb55443299f41d25e3dc513d -> Peer 7b496d70bc7c49da9032416864dc8ff5 (127.22.225.188:34529): Couldn't send request to peer 7b496d70bc7c49da9032416864dc8ff5. Status: Network error: Client connection negotiation failed: client connection to 127.22.225.188:34529: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 06:28:48.260192 30098 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0c07c7b5cb55443299f41d25e3dc513d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f15b6ec81d284f2baa79fa54469b144b" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 35431 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260928 06:28:48.260815 30097 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0c07c7b5cb55443299f41d25e3dc513d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f15b6ec81d284f2baa79fa54469b144b (127.22.225.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0c07c7b5cb55443299f41d25e3dc513d" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 41857 } } peers { permanent_uuid: "f15b6ec81d284f2baa79fa54469b144b" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 35431 } attrs { promote: false } } peers { permanent_uuid: "7b496d70bc7c49da9032416864dc8ff5" member_type: NON_VOTER last_known_addr { host: "127.22.225.188" port: 34529 } attrs { promote: true } } }
I20260928 06:28:48.260957 30147 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f15b6ec81d284f2baa79fa54469b144b (127.22.225.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0c07c7b5cb55443299f41d25e3dc513d" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 41857 } } peers { permanent_uuid: "f15b6ec81d284f2baa79fa54469b144b" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 35431 } attrs { promote: false } } peers { permanent_uuid: "7b496d70bc7c49da9032416864dc8ff5" member_type: NON_VOTER last_known_addr { host: "127.22.225.188" port: 34529 } attrs { promote: true } } }
I20260928 06:28:48.261746 30097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c07c7b5cb55443299f41d25e3dc513d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0c07c7b5cb55443299f41d25e3dc513d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0c07c7b5cb55443299f41d25e3dc513d" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 41857 } } peers { permanent_uuid: "f15b6ec81d284f2baa79fa54469b144b" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 35431 } attrs { promote: false } } peers { permanent_uuid: "7b496d70bc7c49da9032416864dc8ff5" member_type: NON_VOTER last_known_addr { host: "127.22.225.188" port: 34529 } attrs { promote: true } } }
I20260928 06:28:48.261755 30176 sys_catalog.cc:455] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0c07c7b5cb55443299f41d25e3dc513d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0c07c7b5cb55443299f41d25e3dc513d" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 41857 } } peers { permanent_uuid: "f15b6ec81d284f2baa79fa54469b144b" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 35431 } attrs { promote: false } } peers { permanent_uuid: "7b496d70bc7c49da9032416864dc8ff5" member_type: NON_VOTER last_known_addr { host: "127.22.225.188" port: 34529 } attrs { promote: true } } }
I20260928 06:28:48.261822 30097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c07c7b5cb55443299f41d25e3dc513d [sys.catalog]: This master's current role is: LEADER
I20260928 06:28:48.261833 30176 sys_catalog.cc:458] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:28:48.261883 30097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c07c7b5cb55443299f41d25e3dc513d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0c07c7b5cb55443299f41d25e3dc513d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0c07c7b5cb55443299f41d25e3dc513d" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 41857 } } peers { permanent_uuid: "f15b6ec81d284f2baa79fa54469b144b" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 35431 } attrs { promote: false } } peers { permanent_uuid: "7b496d70bc7c49da9032416864dc8ff5" member_type: NON_VOTER last_known_addr { host: "127.22.225.188" port: 34529 } attrs { promote: true } } }
I20260928 06:28:48.261945 30097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c07c7b5cb55443299f41d25e3dc513d [sys.catalog]: This master's current role is: LEADER
I20260928 06:28:48.267562 30191 master.cc:552] Master@127.22.225.188:34529 shutdown complete.
I20260928 06:28:48.267616 30191 master_runner.cc:305] Clearing existing system tablet
I20260928 06:28:48.267747 30191 fs_manager.cc:359] Metadata directory not provided
I20260928 06:28:48.267784 30191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:28:48.268565 30191 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:28:48.269124 30288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:48.269318 30191 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:28:48.269379 30191 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:28:48.269402 30191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/master-2/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/master-2/wal
uuid: "7b496d70bc7c49da9032416864dc8ff5"
format_stamp: "Formatted at 2026-09-28 06:28:48 on dist-test-slave-d59m"
server_key: "9fc870c68ceb2f0681391b3c99dd4c69"
server_key_iv: "358f4d54f4175617b22a59f90d7e60d1"
server_key_version: "encryptionkey@0"
I20260928 06:28:48.270110 30191 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 7b496d70bc7c49da9032416864dc8ff5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:28:48.271744 30191 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 7b496d70bc7c49da9032416864dc8ff5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 06:28:48.271806 30191 log.cc:1196] T 00000000000000000000000000000000 P 7b496d70bc7c49da9032416864dc8ff5: Deleting WAL directory at /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 06:28:48.272007 30191 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 7b496d70bc7c49da9032416864dc8ff5: Deleting consensus metadata
I20260928 06:28:48.272123 30191 master_runner.cc:315] Copying system tablet from 127.22.225.190:41857
I20260928 06:28:48.272939 30191 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 7b496d70bc7c49da9032416864dc8ff5: tablet copy: Beginning tablet copy session from remote peer at address 127.22.225.190:41857
I20260928 06:28:48.283988 29578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:48.273359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53752 (local address 127.22.225.190:41857)
0928 06:28:48.273413 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:48.273416 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:48.276806 (+ 3390us) server_negotiation.cc:407] Connection header received
0928 06:28:48.276937 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:48.276939 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:48.276953 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:48.276989 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:48.277517 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.278313 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.279364 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.279662 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.279703 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:48.280910 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:48.280912 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:48.280913 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:48.280920 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:48.282710 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:48.283168 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:48.283169 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:48.283169 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:48.283226 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:48.283412 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:48.283412 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:48.283413 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:48.283539 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:48.283580 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:48.283787 (+ 207us) server_negotiation.cc:299] Negotiation successful
0928 06:28:48.283893 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 06:28:48.284677 30294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:48.273302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:41857 (local address 127.0.0.1:53752)
0928 06:28:48.276701 (+ 3399us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:48.276718 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:48.276844 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:48.277084 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:48.277086 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:48.277328 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:48.277399 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:48.277401 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.278420 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.278421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:48.279263 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:48.279265 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.279786 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.279787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:48.279849 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:48.280381 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:48.280387 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:48.280809 (+ 422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:48.282869 (+ 2060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:48.282870 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:48.282870 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:48.283087 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:48.283292 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:48.283292 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:48.283293 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:48.283347 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:48.283625 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:48.283626 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:48.283672 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 06:28:48.284406 (+ 734us) client_negotiation.cc:240] Negotiation successful
0928 06:28:48.284516 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3371,"thread_start_us":203,"threads_started":1}
I20260928 06:28:48.285125 29558 tablet_copy_service.cc:140] P 0c07c7b5cb55443299f41d25e3dc513d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 7b496d70bc7c49da9032416864dc8ff5 ({username='oryx', principal='oryx/127.22.225.188@KRBTEST.COM'} at 127.0.0.1:53752)
I20260928 06:28:48.285181 29558 tablet_copy_service.cc:161] P 0c07c7b5cb55443299f41d25e3dc513d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 7b496d70bc7c49da9032416864dc8ff5 at {username='oryx', principal='oryx/127.22.225.188@KRBTEST.COM'} at 127.0.0.1:53752: session id = 7b496d70bc7c49da9032416864dc8ff5-00000000000000000000000000000000
I20260928 06:28:48.285758 29558 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 0c07c7b5cb55443299f41d25e3dc513d: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 06:28:48.286401 30191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 06:28:48.287738 30191 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 7b496d70bc7c49da9032416864dc8ff5: tablet copy: Starting download of 0 data blocks...
I20260928 06:28:48.287864 30191 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 7b496d70bc7c49da9032416864dc8ff5: tablet copy: Starting download of 1 WAL segments...
I20260928 06:28:48.288666 30191 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 7b496d70bc7c49da9032416864dc8ff5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 06:28:48.289535 29558 tablet_copy_service.cc:342] P 0c07c7b5cb55443299f41d25e3dc513d: Request end of tablet copy session 7b496d70bc7c49da9032416864dc8ff5-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.22.225.188@KRBTEST.COM'} at 127.0.0.1:53752
I20260928 06:28:48.289583 29558 tablet_copy_service.cc:433] P 0c07c7b5cb55443299f41d25e3dc513d: ending tablet copy session 7b496d70bc7c49da9032416864dc8ff5-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 7b496d70bc7c49da9032416864dc8ff5
I20260928 06:28:48.291934 30191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:28:48.293344 30296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:48.293576 30191 server_base.cc:1034] running on GCE node
W20260928 06:28:48.293736 30295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:48.296717 30298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:48.296964 30191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:28:48.297109 30191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:28:48.298249 30191 hybrid_clock.cc:648] HybridClock initialized: now 1790576928298240 us; error 26 us; skew 500 ppm
Sep 28 06:28:48 dist-test-slave-d59m krb5kdc[29493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576928, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.225.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:28:48.299748 30191 init.cc:377] Logged in from keytab as oryx/127.22.225.188@KRBTEST.COM (short username oryx)
I20260928 06:28:48.300004 30191 webserver.cc:460] Webserver started at http://127.22.225.188:42067/ using document root <none> and password file <none>
I20260928 06:28:48.300117 30191 fs_manager.cc:359] Metadata directory not provided
I20260928 06:28:48.300144 30191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:28:48.304819 30191 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.001s
I20260928 06:28:48.305485 30305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:48.305665 30191 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:48.305719 30191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/master-2/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/master-2/wal
uuid: "7b496d70bc7c49da9032416864dc8ff5"
format_stamp: "Formatted at 2026-09-28 06:28:48 on dist-test-slave-d59m"
server_key: "9fc870c68ceb2f0681391b3c99dd4c69"
server_key_iv: "358f4d54f4175617b22a59f90d7e60d1"
server_key_version: "encryptionkey@0"
I20260928 06:28:48.305776 30191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:28:48.313987 30191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:28:48.316864 30191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:28:48.319986 30191 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.188:34529
I20260928 06:28:48.320326 30191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/master-2/data/info.pb
I20260928 06:28:48.321331 30358 sys_catalog.cc:263] Verifying existing consensus state
I20260928 06:28:48.321866 30358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b496d70bc7c49da9032416864dc8ff5: Bootstrap starting.
I20260928 06:28:48.323166 30357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.188:34529 every 8 connection(s)
I20260928 06:28:48.324919 30358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b496d70bc7c49da9032416864dc8ff5: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:28:48.325235 30358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b496d70bc7c49da9032416864dc8ff5: Bootstrap complete.
I20260928 06:28:48.325435 30358 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b496d70bc7c49da9032416864dc8ff5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0c07c7b5cb55443299f41d25e3dc513d" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 41857 } } peers { permanent_uuid: "f15b6ec81d284f2baa79fa54469b144b" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 35431 } attrs { promote: false } } peers { permanent_uuid: "7b496d70bc7c49da9032416864dc8ff5" member_type: NON_VOTER last_known_addr { host: "127.22.225.188" port: 34529 } attrs { promote: true } }
I20260928 06:28:48.325536 30358 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b496d70bc7c49da9032416864dc8ff5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7b496d70bc7c49da9032416864dc8ff5, State: Initialized, Role: LEARNER
I20260928 06:28:48.325585 30358 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b496d70bc7c49da9032416864dc8ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0c07c7b5cb55443299f41d25e3dc513d" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 41857 } } peers { permanent_uuid: "f15b6ec81d284f2baa79fa54469b144b" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 35431 } attrs { promote: false } } peers { permanent_uuid: "7b496d70bc7c49da9032416864dc8ff5" member_type: NON_VOTER last_known_addr { host: "127.22.225.188" port: 34529 } attrs { promote: true } }
I20260928 06:28:48.325784 30358 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b496d70bc7c49da9032416864dc8ff5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:28:48.325784 30361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b496d70bc7c49da9032416864dc8ff5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0c07c7b5cb55443299f41d25e3dc513d" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 41857 } } peers { permanent_uuid: "f15b6ec81d284f2baa79fa54469b144b" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 35431 } attrs { promote: false } } peers { permanent_uuid: "7b496d70bc7c49da9032416864dc8ff5" member_type: NON_VOTER last_known_addr { host: "127.22.225.188" port: 34529 } attrs { promote: true } } }
I20260928 06:28:48.326083 30361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b496d70bc7c49da9032416864dc8ff5 [sys.catalog]: This master's current role is: LEARNER
I20260928 06:28:48.327298 30372 catalog_manager.cc:1259] Loaded cluster ID: 3fb00626f39f44199dbf917853eb1818
I20260928 06:28:48.327386 30372 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7b496d70bc7c49da9032416864dc8ff5: loading cluster ID for follower catalog manager: success
I20260928 06:28:48.328562 30372 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7b496d70bc7c49da9032416864dc8ff5: acquiring CA information for follower catalog manager: success
I20260928 06:28:48.328874 30372 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7b496d70bc7c49da9032416864dc8ff5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 06:28:48.344050 30284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:28:48.344311 30284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:28:48.344344 30284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:28:48.346187 30284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:28:48.346246 30284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:28:48.346285 30284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.129
I20260928 06:28:48.348418 30284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:36117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/krb5kdc/oryx_127.22.225.129.keytab
--principal=oryx/127.22.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.129:40749
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.225.129
--webserver_port=40407
--tserver_master_addrs=127.22.225.190:41857,127.22.225.189:35431,127.22.225.188:34529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.129
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:28:48.348888 30284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:28:48.349161 30284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:28:48.354457 30378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:48.354499 30380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:48.354933 30377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:48.355038 30284 server_base.cc:1034] running on GCE node
I20260928 06:28:48.355253 30284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:28:48.355504 30284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:28:48.356649 30284 hybrid_clock.cc:648] HybridClock initialized: now 1790576928356624 us; error 40 us; skew 500 ppm
Sep 28 06:28:48 dist-test-slave-d59m krb5kdc[29493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576928, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:28:48.359272 30284 init.cc:377] Logged in from keytab as oryx/127.22.225.129@KRBTEST.COM (short username oryx)
I20260928 06:28:48.359716 30284 webserver.cc:460] Webserver started at http://127.22.225.129:40407/ using document root <none> and password file <none>
I20260928 06:28:48.359964 30284 fs_manager.cc:359] Metadata directory not provided
I20260928 06:28:48.360049 30284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:28:48.361560 30284 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:28:48.362231 30387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:48.362382 30284 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:28:48.362468 30284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-0/wal
uuid: "c1813e3cd67549a1a161c3c54abc3c3b"
format_stamp: "Formatted at 2026-09-28 06:28:47 on dist-test-slave-d59m"
server_key: "3d6130a81a4088bf3f7160dfbdb64c6d"
server_key_iv: "569e482679cd3450c0a2176c9b856ed3"
server_key_version: "encryptionkey@0"
I20260928 06:28:48.362820 30284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:28:48.379251 30284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:28:48.381660 30284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:28:48.381836 30284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:28:48.382077 30284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:28:48.382388 30284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:28:48.382442 30284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:48.382485 30284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:28:48.382513 30284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:48.389003 30284 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.129:40749
I20260928 06:28:48.389101 30500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.129:40749 every 8 connection(s)
I20260928 06:28:48.389416 30284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
I20260928 06:28:48.392125 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 30284
I20260928 06:28:48.392275 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 29734
Sep 28 06:28:48 dist-test-slave-d59m krb5kdc[29493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576928, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.225.129@KRBTEST.COM for oryx/127.22.225.188@KRBTEST.COM
Sep 28 06:28:48 dist-test-slave-d59m krb5kdc[29493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576928, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.225.129@KRBTEST.COM for oryx/127.22.225.189@KRBTEST.COM
Sep 28 06:28:48 dist-test-slave-d59m krb5kdc[29493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576928, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.225.129@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM
I20260928 06:28:48.400899 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.130:43647
--local_ip_for_outbound_sockets=127.22.225.130
--tserver_master_addrs=127.22.225.190:41857
--webserver_port=36513
--webserver_interface=127.22.225.130
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/krb5kdc/oryx_127.22.225.130.keytab
--principal=oryx/127.22.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:36117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.22.225.190:41857,127.22.225.189:35431
--tserver_master_addrs=127.22.225.190:41857,127.22.225.189:35431,127.22.225.188:34529 with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:28:48.403609 30188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:48.390418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:59041 (local address 127.22.225.189:35431)
0928 06:28:48.390482 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:48.390484 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:48.390825 (+ 341us) server_negotiation.cc:407] Connection header received
0928 06:28:48.391829 (+ 1004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:48.391830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:48.391842 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:48.391874 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:48.394305 (+ 2431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.395044 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.396687 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.396975 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.397014 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:48.399907 (+ 2893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:48.399908 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:48.399909 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:48.399918 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:48.401960 (+ 2042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:48.402400 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:48.402401 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:48.402401 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:48.402463 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:48.402817 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:48.402819 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:48.402820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:48.403003 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:48.403050 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:48.403428 (+ 378us) server_negotiation.cc:299] Negotiation successful
0928 06:28:48.403511 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 06:28:48.404241 30507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:48.390235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.189:35431 (local address 127.22.225.129:59041)
0928 06:28:48.390737 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:48.390747 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:48.391729 (+ 982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:48.392003 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:48.392004 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:48.392506 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:48.392803 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:48.392808 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.395145 (+ 2337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.395146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:48.395912 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:48.395914 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.397099 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.397100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:48.397154 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:48.397973 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:48.397978 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:48.399787 (+ 1809us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:48.402060 (+ 2273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:48.402067 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:48.402069 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:48.402299 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:48.402550 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:48.402551 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:48.402552 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:48.402718 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:48.403104 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:48.403111 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:48.403271 (+ 160us) client_negotiation.cc:769] Sending connection context
0928 06:28:48.403756 (+ 485us) client_negotiation.cc:240] Negotiation successful
0928 06:28:48.403954 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"mutex_wait_us":10,"thread_start_us":343,"threads_started":1}
I20260928 06:28:48.404775 30503 heartbeater.cc:347] Connected to a master server at 127.22.225.189:35431
I20260928 06:28:48.404876 30503 heartbeater.cc:464] Registering TS with master...
I20260928 06:28:48.405057 30503 heartbeater.cc:511] Master 127.22.225.189:35431 requested a full tablet report, sending...
I20260928 06:28:48.405392 29578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:48.390994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:53233 (local address 127.22.225.190:41857)
0928 06:28:48.391041 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:48.391043 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:48.391051 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:28:48.391829 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:48.391830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:48.391842 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:48.391874 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:48.393116 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.393844 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.396654 (+ 2810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.396951 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.397014 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:48.401355 (+ 4341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:48.401357 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:48.401357 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:48.401365 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:48.404075 (+ 2710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:48.404469 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:48.404470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:48.404470 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:48.404524 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:48.404770 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:48.404771 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:48.404772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:48.404914 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:48.404974 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:48.405182 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 06:28:48.405273 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:28:48.405767 30508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:48.390803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:41857 (local address 127.22.225.129:53233)
0928 06:28:48.390917 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:48.390925 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:48.391729 (+ 804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:48.392003 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:48.392004 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:48.392949 (+ 945us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:48.393006 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:48.393007 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.393962 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.393963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:48.396524 (+ 2561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:48.396527 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.397073 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.397074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:48.397141 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:48.399451 (+ 2310us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:48.399457 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:48.400915 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:48.404165 (+ 3250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:48.404166 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:48.404167 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:48.404388 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:48.404629 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:48.404630 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:48.404630 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:48.404697 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:48.405008 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:48.405009 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:48.405053 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 06:28:48.405484 (+ 431us) client_negotiation.cc:240] Negotiation successful
0928 06:28:48.405619 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"mutex_wait_us":1723,"thread_start_us":62,"threads_started":1}
I20260928 06:28:48.406172 30502 heartbeater.cc:347] Connected to a master server at 127.22.225.190:41857
I20260928 06:28:48.406217 30502 heartbeater.cc:464] Registering TS with master...
I20260928 06:28:48.406278 30502 heartbeater.cc:511] Master 127.22.225.190:41857 requested a full tablet report, sending...
I20260928 06:28:48.406598 29538 ts_manager.cc:194] Re-registered known tserver with Master: c1813e3cd67549a1a161c3c54abc3c3b (127.22.225.129:40749)
I20260928 06:28:48.406716 30137 ts_manager.cc:194] Registered new tserver with Master: c1813e3cd67549a1a161c3c54abc3c3b (127.22.225.129:40749)
I20260928 06:28:48.406924 29538 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.225.129@KRBTEST.COM'} at 127.22.225.129:53233
I20260928 06:28:48.408517 30505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:48.390326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:44315 (local address 127.22.225.188:34529)
0928 06:28:48.390450 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:48.390452 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:48.390801 (+ 349us) server_negotiation.cc:407] Connection header received
0928 06:28:48.391819 (+ 1018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:48.391820 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:48.391835 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:48.391923 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:48.392891 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.393957 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.395400 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.395747 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.395790 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:48.401669 (+ 5879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:48.401671 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:48.401671 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:48.401679 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:48.407269 (+ 5590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:48.407678 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:48.407679 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:48.407679 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:48.407738 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:48.407915 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:48.407915 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:48.407915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:48.408071 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:48.408117 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:48.408313 (+ 196us) server_negotiation.cc:299] Negotiation successful
0928 06:28:48.408404 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":51,"threads_started":1}
I20260928 06:28:48.409001 30506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:48.390234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.188:34529 (local address 127.22.225.129:44315)
0928 06:28:48.390655 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:48.390701 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:48.391708 (+ 1007us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:48.391999 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:48.392004 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:48.392516 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:48.392803 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:48.392809 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.394044 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.394045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:48.395309 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:48.395311 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.395878 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.395878 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:48.396023 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:48.397668 (+ 1645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:48.397680 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:48.401244 (+ 3564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:48.407359 (+ 6115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:48.407360 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:48.407360 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:48.407588 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:48.407804 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:48.407805 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:48.407805 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:48.407855 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:48.408169 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:48.408171 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:48.408208 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 06:28:48.408754 (+ 546us) client_negotiation.cc:240] Negotiation successful
0928 06:28:48.408877 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"mutex_wait_us":48,"thread_start_us":346,"threads_started":1}
I20260928 06:28:48.409706 30501 heartbeater.cc:347] Connected to a master server at 127.22.225.188:34529
I20260928 06:28:48.409754 30501 heartbeater.cc:464] Registering TS with master...
I20260928 06:28:48.409822 30501 heartbeater.cc:511] Master 127.22.225.188:34529 requested a full tablet report, sending...
I20260928 06:28:48.410228 30322 ts_manager.cc:194] Registered new tserver with Master: c1813e3cd67549a1a161c3c54abc3c3b (127.22.225.129:40749)
W20260928 06:28:48.515741 30509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:28:48.515949 30509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:28:48.515971 30509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:28:48.517380 30509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:28:48.517434 30509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:28:48.517482 30509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.130
I20260928 06:28:48.519025 30509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:36117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/krb5kdc/oryx_127.22.225.130.keytab
--principal=oryx/127.22.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.130:43647
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.225.130
--webserver_port=36513
--tserver_master_addrs=127.22.225.190:41857,127.22.225.189:35431,127.22.225.188:34529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.130
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:28:48.519273 30509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:28:48.519523 30509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:28:48.524556 30514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:48.524679 30509 server_base.cc:1034] running on GCE node
W20260928 06:28:48.524763 30515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:48.524547 30517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:48.525020 30509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:28:48.525367 30509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:28:48.526544 30509 hybrid_clock.cc:648] HybridClock initialized: now 1790576928526529 us; error 46 us; skew 500 ppm
Sep 28 06:28:48 dist-test-slave-d59m krb5kdc[29493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576928, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:28:48.529208 30509 init.cc:377] Logged in from keytab as oryx/127.22.225.130@KRBTEST.COM (short username oryx)
I20260928 06:28:48.529704 30509 webserver.cc:460] Webserver started at http://127.22.225.130:36513/ using document root <none> and password file <none>
I20260928 06:28:48.529989 30509 fs_manager.cc:359] Metadata directory not provided
I20260928 06:28:48.530078 30509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:28:48.531458 30509 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:28:48.532109 30524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:48.532282 30509 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:28:48.532385 30509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-1/wal
uuid: "3a7683a995ae4403b291afc021e4e0d3"
format_stamp: "Formatted at 2026-09-28 06:28:47 on dist-test-slave-d59m"
server_key: "9129e90b6ac80f7cd44caeb89a51fcd7"
server_key_iv: "806da175cb7892f04dce3356107b8449"
server_key_version: "encryptionkey@0"
I20260928 06:28:48.532831 30509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:28:48.554194 30509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:28:48.556674 30509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:28:48.556840 30509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:28:48.557099 30509 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:28:48.557446 30509 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:28:48.557507 30509 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:48.557550 30509 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:28:48.557597 30509 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:48.564072 30509 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.130:43647
I20260928 06:28:48.564143 30637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.130:43647 every 8 connection(s)
I20260928 06:28:48.564568 30509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
I20260928 06:28:48.567022 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 30509
I20260928 06:28:48.567185 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 29868
Sep 28 06:28:48 dist-test-slave-d59m krb5kdc[29493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576928, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.225.130@KRBTEST.COM for oryx/127.22.225.188@KRBTEST.COM
Sep 28 06:28:48 dist-test-slave-d59m krb5kdc[29493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576928, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.225.130@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM
I20260928 06:28:48.575870 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.131:38727
--local_ip_for_outbound_sockets=127.22.225.131
--tserver_master_addrs=127.22.225.190:41857
--webserver_port=37481
--webserver_interface=127.22.225.131
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/krb5kdc/oryx_127.22.225.131.keytab
--principal=oryx/127.22.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:36117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.22.225.190:41857,127.22.225.189:35431
--tserver_master_addrs=127.22.225.190:41857,127.22.225.189:35431,127.22.225.188:34529 with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 06:28:48 dist-test-slave-d59m krb5kdc[29493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576928, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.225.130@KRBTEST.COM for oryx/127.22.225.189@KRBTEST.COM
I20260928 06:28:48.579260 30505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:48.565382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:40053 (local address 127.22.225.188:34529)
0928 06:28:48.565454 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:48.565456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:48.565680 (+ 224us) server_negotiation.cc:407] Connection header received
0928 06:28:48.566372 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:48.566374 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:48.566391 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:48.566422 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:48.567298 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.568060 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.569650 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.570015 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.570067 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:48.575093 (+ 5026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:48.575095 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:48.575095 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:48.575104 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:48.576919 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:48.577744 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:48.577745 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:48.577745 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:48.577797 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:48.578093 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:48.578094 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:48.578094 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:48.578226 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:48.578294 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:48.579015 (+ 721us) server_negotiation.cc:299] Negotiation successful
0928 06:28:48.579113 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 06:28:48.581306 29578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:48.565909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:59389 (local address 127.22.225.190:41857)
0928 06:28:48.565953 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:48.565955 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:48.565962 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:28:48.566354 (+ 392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:48.566355 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:48.566368 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:48.566424 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:48.567498 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.568301 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.569634 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.569961 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.570016 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:48.577050 (+ 7034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:48.577052 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:48.577052 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:48.577062 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:48.579918 (+ 2856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:48.580340 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:48.580341 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:48.580342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:48.580401 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:48.580667 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:48.580668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:48.580668 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:48.580815 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:48.580863 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:48.581083 (+ 220us) server_negotiation.cc:299] Negotiation successful
0928 06:28:48.581177 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:28:48.581496 30188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:48.571063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:53859 (local address 127.22.225.189:35431)
0928 06:28:48.571115 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:48.571117 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:48.571222 (+ 105us) server_negotiation.cc:407] Connection header received
0928 06:28:48.571268 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:48.571269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:48.571279 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:48.571332 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:48.571694 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.572463 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.573570 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.573915 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.573955 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:48.576944 (+ 2989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:48.576945 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:48.576945 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:48.576954 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:48.580139 (+ 3185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:48.580594 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:48.580596 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:48.580597 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:48.580658 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:48.580873 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:48.580874 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:48.580874 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:48.581031 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:48.581080 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:48.581306 (+ 226us) server_negotiation.cc:299] Negotiation successful
0928 06:28:48.581393 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 06:28:48.581945 30642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:48.565344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.188:34529 (local address 127.22.225.130:40053)
0928 06:28:48.565542 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:48.565590 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:48.566244 (+ 654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:48.566517 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:48.566521 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:48.566888 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:48.567168 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:48.567173 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.568204 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.568204 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:48.569524 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:48.569541 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.570157 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.570158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:48.570261 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:48.571523 (+ 1262us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:48.571541 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:48.574965 (+ 3424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:48.577330 (+ 2365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:48.577337 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:48.577340 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:48.577661 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:48.577873 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:48.577874 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:48.577874 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:48.578025 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:48.578304 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:48.578311 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:48.578481 (+ 170us) client_negotiation.cc:769] Sending connection context
0928 06:28:48.578995 (+ 514us) client_negotiation.cc:240] Negotiation successful
0928 06:28:48.581641 (+ 2646us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":2,"thread_start_us":90,"threads_started":1}
I20260928 06:28:48.581961 30644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:48.570979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.189:35431 (local address 127.22.225.130:53859)
0928 06:28:48.571141 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:48.571151 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:48.571214 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:48.571356 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:48.571357 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:48.571554 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:48.571622 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:48.571624 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.572558 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.572559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:48.573466 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:48.573469 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.574040 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.574041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:48.574111 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:48.575288 (+ 1177us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:48.575294 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:48.576838 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:48.580241 (+ 3403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:48.580242 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:48.580243 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:48.580479 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:48.580739 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:48.580740 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:48.580740 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:48.580799 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:48.581135 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:48.581136 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:48.581183 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 06:28:48.581570 (+ 387us) client_negotiation.cc:240] Negotiation successful
0928 06:28:48.581692 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":109,"threads_started":1}
I20260928 06:28:48.582531 30638 heartbeater.cc:347] Connected to a master server at 127.22.225.188:34529
I20260928 06:28:48.582535 30640 heartbeater.cc:347] Connected to a master server at 127.22.225.189:35431
I20260928 06:28:48.582646 30640 heartbeater.cc:464] Registering TS with master...
I20260928 06:28:48.582646 30638 heartbeater.cc:464] Registering TS with master...
I20260928 06:28:48.582863 30638 heartbeater.cc:511] Master 127.22.225.188:34529 requested a full tablet report, sending...
I20260928 06:28:48.582863 30640 heartbeater.cc:511] Master 127.22.225.189:35431 requested a full tablet report, sending...
I20260928 06:28:48.582952 30643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:48.565703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:41857 (local address 127.22.225.130:59389)
0928 06:28:48.565877 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:48.565888 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:48.566256 (+ 368us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:48.566517 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:48.566521 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:48.567319 (+ 798us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:48.567392 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:48.567394 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.568404 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.568405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:48.569527 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:48.569529 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.570108 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.570109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:48.570254 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:48.571735 (+ 1481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:48.571740 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:48.576083 (+ 4343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:48.580018 (+ 3935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:48.580019 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:48.580020 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:48.580248 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:48.580478 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:48.580478 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:48.580479 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:48.580551 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:48.580914 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:48.580915 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:48.580962 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 06:28:48.581435 (+ 473us) client_negotiation.cc:240] Negotiation successful
0928 06:28:48.582598 (+ 1163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":1031,"thread_start_us":79,"threads_started":1}
I20260928 06:28:48.583328 30137 ts_manager.cc:194] Registered new tserver with Master: 3a7683a995ae4403b291afc021e4e0d3 (127.22.225.130:43647)
I20260928 06:28:48.583393 30639 heartbeater.cc:347] Connected to a master server at 127.22.225.190:41857
I20260928 06:28:48.583382 30322 ts_manager.cc:194] Registered new tserver with Master: 3a7683a995ae4403b291afc021e4e0d3 (127.22.225.130:43647)
I20260928 06:28:48.583442 30639 heartbeater.cc:464] Registering TS with master...
I20260928 06:28:48.583515 30639 heartbeater.cc:511] Master 127.22.225.190:41857 requested a full tablet report, sending...
I20260928 06:28:48.583832 29538 ts_manager.cc:194] Re-registered known tserver with Master: 3a7683a995ae4403b291afc021e4e0d3 (127.22.225.130:43647)
I20260928 06:28:48.584393 29538 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.225.130@KRBTEST.COM'} at 127.22.225.130:59389
W20260928 06:28:48.671486 30645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:28:48.671759 30645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:28:48.671795 30645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:28:48.673702 30645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:28:48.673763 30645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:28:48.673813 30645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.131
I20260928 06:28:48.676311 30645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:36117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/krb5kdc/oryx_127.22.225.131.keytab
--principal=oryx/127.22.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.131:38727
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.225.131
--webserver_port=37481
--tserver_master_addrs=127.22.225.190:41857,127.22.225.189:35431,127.22.225.188:34529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.131
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:28:48.676682 30645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:28:48.676939 30645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:28:48.683243 30650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:48.683388 30645 server_base.cc:1034] running on GCE node
W20260928 06:28:48.683308 30653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:48.683259 30651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:48.683694 30645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:28:48.683979 30645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:28:48.685139 30645 hybrid_clock.cc:648] HybridClock initialized: now 1790576928685118 us; error 37 us; skew 500 ppm
Sep 28 06:28:48 dist-test-slave-d59m krb5kdc[29493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576928, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:28:48.687734 30645 init.cc:377] Logged in from keytab as oryx/127.22.225.131@KRBTEST.COM (short username oryx)
I20260928 06:28:48.688190 30645 webserver.cc:460] Webserver started at http://127.22.225.131:37481/ using document root <none> and password file <none>
I20260928 06:28:48.688424 30645 fs_manager.cc:359] Metadata directory not provided
I20260928 06:28:48.688478 30645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:28:48.689904 30645 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:28:48.690543 30660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:48.690754 30645 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:28:48.690853 30645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-2/wal
uuid: "377df16ac26d45a8b8114cc6d81d4894"
format_stamp: "Formatted at 2026-09-28 06:28:47 on dist-test-slave-d59m"
server_key: "adc62e2666ea8a27a7999655e1e533e0"
server_key_iv: "860d90871f7420df1624b7d756cb49f9"
server_key_version: "encryptionkey@0"
I20260928 06:28:48.691243 30645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:28:48.699667 30645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:28:48.702280 30645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:28:48.702430 30645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:28:48.702662 30645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:28:48.703027 30645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:28:48.703063 30645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:48.703085 30645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:28:48.703097 30645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:48.709913 30645 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.131:38727
I20260928 06:28:48.709965 30773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.131:38727 every 8 connection(s)
I20260928 06:28:48.710851 30645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
I20260928 06:28:48.711490 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 30645
Sep 28 06:28:48 dist-test-slave-d59m krb5kdc[29493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576928, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.225.131@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM
Sep 28 06:28:48 dist-test-slave-d59m krb5kdc[29493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576928, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.225.131@KRBTEST.COM for oryx/127.22.225.189@KRBTEST.COM
I20260928 06:28:48.723614 30788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:48.713419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44636 (local address 127.22.225.189:35431)
0928 06:28:48.713546 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:48.713548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:48.713797 (+ 249us) server_negotiation.cc:407] Connection header received
0928 06:28:48.713890 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:48.713892 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:48.713904 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:48.713988 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:48.714766 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.715534 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.716707 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.717007 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.717058 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:48.719031 (+ 1973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:48.719033 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:48.719033 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:48.719039 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:48.722223 (+ 3184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:48.722609 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:48.722610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:48.722610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:48.722662 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:48.722825 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:48.722826 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:48.722826 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:48.722968 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:48.723056 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:48.723406 (+ 350us) server_negotiation.cc:299] Negotiation successful
0928 06:28:48.723484 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":62,"threads_started":1}
I20260928 06:28:48.725049 30785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:48.713089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53760 (local address 127.22.225.190:41857)
0928 06:28:48.713643 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:48.713645 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:48.713821 (+ 176us) server_negotiation.cc:407] Connection header received
0928 06:28:48.713853 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:48.713854 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:48.713868 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:48.713940 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:48.714854 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.715653 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.716744 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.717028 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.717070 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:48.719395 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:48.719396 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:48.719396 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:48.719403 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:48.722069 (+ 2666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:48.724121 (+ 2052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:48.724122 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:48.724123 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:48.724184 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:48.724443 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:48.724444 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:48.724444 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:48.724618 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:48.724716 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:48.724832 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 06:28:48.724941 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":88,"threads_started":1}
I20260928 06:28:48.727589 30784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:48.713036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.189:35431 (local address 127.22.225.131:56651)
0928 06:28:48.713222 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:48.713230 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:48.714634 (+ 1404us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:48.714922 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:48.714923 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:48.715218 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:48.715488 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:48.715493 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.716371 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.716372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:48.717553 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:48.717555 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.718332 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.718333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:48.718473 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:48.720772 (+ 2299us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:48.720778 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:48.722615 (+ 1837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:48.725958 (+ 3343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:48.725964 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:48.725967 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:48.726246 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:48.726473 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:48.726474 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:48.726474 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:48.726607 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:48.726893 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:48.726899 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:48.727058 (+ 159us) client_negotiation.cc:769] Sending connection context
0928 06:28:48.727179 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 06:28:48.727337 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":85,"threads_started":1}
I20260928 06:28:48.727801 30789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:48.713454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42208 (local address 127.22.225.188:34529)
0928 06:28:48.713690 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:48.713692 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:48.713965 (+ 273us) server_negotiation.cc:407] Connection header received
0928 06:28:48.714067 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:48.714068 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:48.714081 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:48.714119 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:48.717680 (+ 3561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.718409 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.719632 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.719933 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.720003 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:48.723770 (+ 3767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:48.723771 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:48.723771 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:48.723778 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:48.726419 (+ 2641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:48.726853 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:48.726855 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:48.726855 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:48.726909 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:48.727235 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:48.727236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:48.727236 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:48.727354 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:48.727419 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:48.727618 (+ 199us) server_negotiation.cc:299] Negotiation successful
0928 06:28:48.727705 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":52,"threads_started":1}
I20260928 06:28:48.728060 29578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:48.712857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:54229 (local address 127.22.225.190:41857)
0928 06:28:48.712947 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:48.712949 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:48.713154 (+ 205us) server_negotiation.cc:407] Connection header received
0928 06:28:48.714714 (+ 1560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:48.714715 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:48.714728 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:48.714777 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:48.716434 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.717134 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.718012 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.718279 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.718341 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:48.723210 (+ 4869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:48.723211 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:48.723211 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:48.723218 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:48.726722 (+ 3504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:48.727120 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:48.727121 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:48.727121 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:48.727184 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:48.727351 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:48.727352 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:48.727352 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:48.727456 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:48.727509 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:48.727894 (+ 385us) server_negotiation.cc:299] Negotiation successful
0928 06:28:48.727962 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Sep 28 06:28:48 dist-test-slave-d59m krb5kdc[29493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576928, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.225.131@KRBTEST.COM for oryx/127.22.225.188@KRBTEST.COM
I20260928 06:28:48.728775 30188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:48.713107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:56651 (local address 127.22.225.189:35431)
0928 06:28:48.713166 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:48.713169 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:48.713546 (+ 377us) server_negotiation.cc:407] Connection header received
0928 06:28:48.714823 (+ 1277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:48.714824 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:48.714835 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:48.714881 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:48.715562 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.716259 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.717632 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.717914 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.717988 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:48.722721 (+ 4733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:48.722722 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:48.722723 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:48.722731 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:48.725713 (+ 2982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:48.726336 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:48.726337 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:48.726337 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:48.726392 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:48.726682 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:48.726683 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:48.726684 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:48.726809 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:48.726866 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:48.728594 (+ 1728us) server_negotiation.cc:299] Negotiation successful
0928 06:28:48.728692 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 06:28:48.727821 30783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:48.712763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:41857 (local address 127.22.225.131:54229)
0928 06:28:48.713079 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:48.713090 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:48.714494 (+ 1404us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:48.715020 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:48.715020 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:48.715634 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:48.715689 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:48.715690 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.717218 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.717219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:48.717929 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:48.717931 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.718394 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.718395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:48.718480 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:48.720104 (+ 1624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:48.720113 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:48.723111 (+ 2998us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:48.726837 (+ 3726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:48.726838 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:48.726838 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:48.727044 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:48.727244 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:48.727245 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:48.727245 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:48.727294 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:48.727532 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:48.727532 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:48.727569 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 06:28:48.727626 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 06:28:48.727714 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":122,"threads_started":1}
I20260928 06:28:48.729668 30776 heartbeater.cc:347] Connected to a master server at 127.22.225.189:35431
I20260928 06:28:48.729749 30776 heartbeater.cc:464] Registering TS with master...
I20260928 06:28:48.729920 30776 heartbeater.cc:511] Master 127.22.225.189:35431 requested a full tablet report, sending...
I20260928 06:28:48.730046 30775 heartbeater.cc:347] Connected to a master server at 127.22.225.190:41857
I20260928 06:28:48.730105 30775 heartbeater.cc:464] Registering TS with master...
I20260928 06:28:48.730165 30775 heartbeater.cc:511] Master 127.22.225.190:41857 requested a full tablet report, sending...
I20260928 06:28:48.730391 30137 ts_manager.cc:194] Registered new tserver with Master: 377df16ac26d45a8b8114cc6d81d4894 (127.22.225.131:38727)
I20260928 06:28:48.730598 29537 ts_manager.cc:194] Re-registered known tserver with Master: 377df16ac26d45a8b8114cc6d81d4894 (127.22.225.131:38727)
I20260928 06:28:48.728729 29538 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53760:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:28:48.731004 29538 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:28:48.731015 29537 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.225.131@KRBTEST.COM'} at 127.22.225.131:54229
I20260928 06:28:48.732733 30505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:48.712427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:34639 (local address 127.22.225.188:34529)
0928 06:28:48.712473 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:48.712475 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:48.712785 (+ 310us) server_negotiation.cc:407] Connection header received
0928 06:28:48.714609 (+ 1824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:48.714610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:48.714621 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:48.714652 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:48.718915 (+ 4263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.719512 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.723005 (+ 3493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.723286 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.723328 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:48.728710 (+ 5382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:48.728712 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:48.728712 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:48.728718 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:48.731405 (+ 2687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:48.731943 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:48.731944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:48.731944 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:48.732021 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:48.732224 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:48.732224 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:48.732225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:48.732326 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:48.732398 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:48.732550 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 06:28:48.732642 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 06:28:48.733031 30781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:48.712419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.188:34529 (local address 127.22.225.131:34639)
0928 06:28:48.712638 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:48.712688 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:48.714485 (+ 1797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:48.714793 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:48.714796 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:48.718762 (+ 3966us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:48.718822 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:48.718824 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.719610 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.719611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:48.722897 (+ 3286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:48.722899 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.723408 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.723409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:48.723486 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:48.724758 (+ 1272us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:48.724763 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:48.728617 (+ 3854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:48.731499 (+ 2882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:48.731500 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:48.731500 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:48.731868 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:48.732085 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:48.732086 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:48.732086 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:48.732150 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:48.732391 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:48.732392 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:48.732430 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 06:28:48.732515 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 06:28:48.732641 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":97,"threads_started":1}
I20260928 06:28:48.733753 30774 heartbeater.cc:347] Connected to a master server at 127.22.225.188:34529
I20260928 06:28:48.733811 30774 heartbeater.cc:464] Registering TS with master...
I20260928 06:28:48.733884 30774 heartbeater.cc:511] Master 127.22.225.188:34529 requested a full tablet report, sending...
I20260928 06:28:48.734160 30322 ts_manager.cc:194] Registered new tserver with Master: 377df16ac26d45a8b8114cc6d81d4894 (127.22.225.131:38727)
I20260928 06:28:48.739028 30795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:48.735010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.129:40749 (local address 127.0.0.1:35048)
0928 06:28:48.735422 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:48.735430 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:48.735479 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:48.735773 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:48.735774 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:48.735776 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:28:48.735815 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:48.735817 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.736969 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.736970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:48.737826 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:48.737828 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.738821 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.738822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:48.738881 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:48.738885 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:28:48.738926 (+ 41us) client_negotiation.cc:240] Negotiation successful
0928 06:28:48.738963 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259}
I20260928 06:28:48.739056 30796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:48.735164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35048 (local address 127.22.225.129:40749)
0928 06:28:48.735363 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:48.735368 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:48.735493 (+ 125us) server_negotiation.cc:407] Connection header received
0928 06:28:48.735598 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:48.735601 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:48.735619 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:48.735700 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:28:48.735892 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.736874 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.737920 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.738719 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.738840 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:48.738899 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:48.738938 (+ 39us) server_negotiation.cc:299] Negotiation successful
0928 06:28:48.738983 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":124,"threads_started":1}
I20260928 06:28:48.739172 30096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:48.735127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.131:38727 (local address 127.0.0.1:58974)
0928 06:28:48.735300 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:48.735313 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:48.735359 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:48.735735 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:48.735736 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:48.735737 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:28:48.735782 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:48.735784 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.737165 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.737166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:48.738173 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:48.738175 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.738930 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.738931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:48.738991 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:48.738995 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:28:48.739024 (+ 29us) client_negotiation.cc:240] Negotiation successful
0928 06:28:48.739072 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":119,"threads_started":1}
I20260928 06:28:48.739971 30797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:48.735388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58974 (local address 127.22.225.131:38727)
0928 06:28:48.735571 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:48.735576 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:48.735586 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:28:48.735619 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:48.735621 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:48.735636 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:48.735854 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:28:48.735898 (+ 44us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.737070 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.738276 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.738835 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.739770 (+ 935us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:48.739828 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:48.739865 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 06:28:48.739899 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":100,"threads_started":1}
I20260928 06:28:48.740031 30798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:48.735628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.130:43647 (local address 127.0.0.1:60230)
0928 06:28:48.735752 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:48.735763 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:48.735804 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:48.736276 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:48.736277 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:48.736278 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:28:48.736313 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:48.736315 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.737471 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.737472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:48.738499 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:48.738502 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.739815 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.739816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:48.739889 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:48.739893 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:28:48.739922 (+ 29us) client_negotiation.cc:240] Negotiation successful
0928 06:28:48.739956 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":84,"threads_started":1}
I20260928 06:28:48.740060 30435 tablet_service.cc:1467] Processing CreateTablet for tablet 41aaf57ad53244039bfc792774f52340 (DEFAULT_TABLE table=test-table [id=4b4d779dc532485bb136ad3a5b993b3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:28:48.740468 30435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41aaf57ad53244039bfc792774f52340. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:48.740705 30799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:48.735944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60230 (local address 127.22.225.130:43647)
0928 06:28:48.736112 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:48.736116 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:48.736126 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:28:48.736166 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:48.736168 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:48.736184 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:48.736265 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:28:48.736360 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.737383 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.738630 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.739685 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.740403 (+ 718us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:48.740463 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:48.740489 (+ 26us) server_negotiation.cc:299] Negotiation successful
0928 06:28:48.740566 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":91,"threads_started":1}
I20260928 06:28:48.741269 30572 tablet_service.cc:1467] Processing CreateTablet for tablet 41aaf57ad53244039bfc792774f52340 (DEFAULT_TABLE table=test-table [id=4b4d779dc532485bb136ad3a5b993b3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:28:48.741616 30572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41aaf57ad53244039bfc792774f52340. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:48.744547 30800 tablet_bootstrap.cc:492] T 41aaf57ad53244039bfc792774f52340 P c1813e3cd67549a1a161c3c54abc3c3b: Bootstrap starting.
I20260928 06:28:48.744570 30708 tablet_service.cc:1467] Processing CreateTablet for tablet 41aaf57ad53244039bfc792774f52340 (DEFAULT_TABLE table=test-table [id=4b4d779dc532485bb136ad3a5b993b3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:28:48.744935 30708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41aaf57ad53244039bfc792774f52340. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:48.745172 30801 tablet_bootstrap.cc:492] T 41aaf57ad53244039bfc792774f52340 P 3a7683a995ae4403b291afc021e4e0d3: Bootstrap starting.
I20260928 06:28:48.745496 30800 tablet_bootstrap.cc:654] T 41aaf57ad53244039bfc792774f52340 P c1813e3cd67549a1a161c3c54abc3c3b: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:48.745857 30800 log.cc:824] T 41aaf57ad53244039bfc792774f52340 P c1813e3cd67549a1a161c3c54abc3c3b: Log is configured to *not* fsync() on all Append() calls
I20260928 06:28:48.747282 30801 tablet_bootstrap.cc:654] T 41aaf57ad53244039bfc792774f52340 P 3a7683a995ae4403b291afc021e4e0d3: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:48.747601 30801 log.cc:824] T 41aaf57ad53244039bfc792774f52340 P 3a7683a995ae4403b291afc021e4e0d3: Log is configured to *not* fsync() on all Append() calls
I20260928 06:28:48.747766 30800 tablet_bootstrap.cc:492] T 41aaf57ad53244039bfc792774f52340 P c1813e3cd67549a1a161c3c54abc3c3b: No bootstrap required, opened a new log
I20260928 06:28:48.747841 30800 ts_tablet_manager.cc:1397] T 41aaf57ad53244039bfc792774f52340 P c1813e3cd67549a1a161c3c54abc3c3b: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260928 06:28:48.748406 30801 tablet_bootstrap.cc:492] T 41aaf57ad53244039bfc792774f52340 P 3a7683a995ae4403b291afc021e4e0d3: No bootstrap required, opened a new log
I20260928 06:28:48.748497 30804 tablet_bootstrap.cc:492] T 41aaf57ad53244039bfc792774f52340 P 377df16ac26d45a8b8114cc6d81d4894: Bootstrap starting.
I20260928 06:28:48.748715 30801 ts_tablet_manager.cc:1397] T 41aaf57ad53244039bfc792774f52340 P 3a7683a995ae4403b291afc021e4e0d3: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 06:28:48.749101 30804 tablet_bootstrap.cc:654] T 41aaf57ad53244039bfc792774f52340 P 377df16ac26d45a8b8114cc6d81d4894: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:48.749379 30804 log.cc:824] T 41aaf57ad53244039bfc792774f52340 P 377df16ac26d45a8b8114cc6d81d4894: Log is configured to *not* fsync() on all Append() calls
I20260928 06:28:48.749871 30800 raft_consensus.cc:348] T 41aaf57ad53244039bfc792774f52340 P c1813e3cd67549a1a161c3c54abc3c3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1813e3cd67549a1a161c3c54abc3c3b" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 40749 } } peers { permanent_uuid: "377df16ac26d45a8b8114cc6d81d4894" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 38727 } } peers { permanent_uuid: "3a7683a995ae4403b291afc021e4e0d3" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 43647 } }
I20260928 06:28:48.750020 30804 tablet_bootstrap.cc:492] T 41aaf57ad53244039bfc792774f52340 P 377df16ac26d45a8b8114cc6d81d4894: No bootstrap required, opened a new log
I20260928 06:28:48.750039 30800 raft_consensus.cc:374] T 41aaf57ad53244039bfc792774f52340 P c1813e3cd67549a1a161c3c54abc3c3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:48.750097 30804 ts_tablet_manager.cc:1397] T 41aaf57ad53244039bfc792774f52340 P 377df16ac26d45a8b8114cc6d81d4894: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:28:48.750080 30800 raft_consensus.cc:729] T 41aaf57ad53244039bfc792774f52340 P c1813e3cd67549a1a161c3c54abc3c3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1813e3cd67549a1a161c3c54abc3c3b, State: Initialized, Role: FOLLOWER
I20260928 06:28:48.750241 30800 consensus_queue.cc:260] T 41aaf57ad53244039bfc792774f52340 P c1813e3cd67549a1a161c3c54abc3c3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1813e3cd67549a1a161c3c54abc3c3b" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 40749 } } peers { permanent_uuid: "377df16ac26d45a8b8114cc6d81d4894" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 38727 } } peers { permanent_uuid: "3a7683a995ae4403b291afc021e4e0d3" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 43647 } }
I20260928 06:28:48.750495 30800 ts_tablet_manager.cc:1428] T 41aaf57ad53244039bfc792774f52340 P c1813e3cd67549a1a161c3c54abc3c3b: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260928 06:28:48.750607 30502 heartbeater.cc:503] Master 127.22.225.190:41857 was elected leader, sending a full tablet report...
I20260928 06:28:48.752415 30804 raft_consensus.cc:348] T 41aaf57ad53244039bfc792774f52340 P 377df16ac26d45a8b8114cc6d81d4894 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1813e3cd67549a1a161c3c54abc3c3b" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 40749 } } peers { permanent_uuid: "377df16ac26d45a8b8114cc6d81d4894" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 38727 } } peers { permanent_uuid: "3a7683a995ae4403b291afc021e4e0d3" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 43647 } }
I20260928 06:28:48.752559 30804 raft_consensus.cc:374] T 41aaf57ad53244039bfc792774f52340 P 377df16ac26d45a8b8114cc6d81d4894 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:48.752615 30804 raft_consensus.cc:729] T 41aaf57ad53244039bfc792774f52340 P 377df16ac26d45a8b8114cc6d81d4894 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 377df16ac26d45a8b8114cc6d81d4894, State: Initialized, Role: FOLLOWER
I20260928 06:28:48.752722 30804 consensus_queue.cc:260] T 41aaf57ad53244039bfc792774f52340 P 377df16ac26d45a8b8114cc6d81d4894 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1813e3cd67549a1a161c3c54abc3c3b" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 40749 } } peers { permanent_uuid: "377df16ac26d45a8b8114cc6d81d4894" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 38727 } } peers { permanent_uuid: "3a7683a995ae4403b291afc021e4e0d3" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 43647 } }
I20260928 06:28:48.752774 30801 raft_consensus.cc:348] T 41aaf57ad53244039bfc792774f52340 P 3a7683a995ae4403b291afc021e4e0d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1813e3cd67549a1a161c3c54abc3c3b" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 40749 } } peers { permanent_uuid: "377df16ac26d45a8b8114cc6d81d4894" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 38727 } } peers { permanent_uuid: "3a7683a995ae4403b291afc021e4e0d3" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 43647 } }
I20260928 06:28:48.752925 30801 raft_consensus.cc:374] T 41aaf57ad53244039bfc792774f52340 P 3a7683a995ae4403b291afc021e4e0d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:48.752962 30801 raft_consensus.cc:729] T 41aaf57ad53244039bfc792774f52340 P 3a7683a995ae4403b291afc021e4e0d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a7683a995ae4403b291afc021e4e0d3, State: Initialized, Role: FOLLOWER
I20260928 06:28:48.752981 30804 ts_tablet_manager.cc:1428] T 41aaf57ad53244039bfc792774f52340 P 377df16ac26d45a8b8114cc6d81d4894: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:28:48.753057 30775 heartbeater.cc:503] Master 127.22.225.190:41857 was elected leader, sending a full tablet report...
I20260928 06:28:48.753084 30801 consensus_queue.cc:260] T 41aaf57ad53244039bfc792774f52340 P 3a7683a995ae4403b291afc021e4e0d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1813e3cd67549a1a161c3c54abc3c3b" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 40749 } } peers { permanent_uuid: "377df16ac26d45a8b8114cc6d81d4894" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 38727 } } peers { permanent_uuid: "3a7683a995ae4403b291afc021e4e0d3" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 43647 } }
I20260928 06:28:48.753337 30801 ts_tablet_manager.cc:1428] T 41aaf57ad53244039bfc792774f52340 P 3a7683a995ae4403b291afc021e4e0d3: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260928 06:28:48.753626 30639 heartbeater.cc:503] Master 127.22.225.190:41857 was elected leader, sending a full tablet report...
I20260928 06:28:48.796224 30505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:48.793165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42214 (local address 127.22.225.188:34529)
0928 06:28:48.793218 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:48.793220 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:48.793233 (+ 13us) server_negotiation.cc:407] Connection header received
0928 06:28:48.793265 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:48.793266 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:48.793294 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:48.793415 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:28:48.793578 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.794133 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.795549 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.795916 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.795996 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:48.796046 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:48.796139 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 06:28:48.796167 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 06:28:48.796257 30798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:48.793084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.188:34529 (local address 127.0.0.1:42214)
0928 06:28:48.793149 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:48.793161 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:48.793209 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:48.793428 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:48.793429 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:48.793431 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:28:48.793496 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:48.793499 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.794203 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.794204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:48.795463 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:48.795466 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:48.796006 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:48.796007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:48.796072 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:48.796075 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:28:48.796129 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 06:28:48.796175 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 06:28:48.796663 30332 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 7b496d70bc7c49da9032416864dc8ff5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260928 06:28:48.797534 30361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b496d70bc7c49da9032416864dc8ff5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c07c7b5cb55443299f41d25e3dc513d. Latest consensus state: current_term: 1 leader_uuid: "0c07c7b5cb55443299f41d25e3dc513d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0c07c7b5cb55443299f41d25e3dc513d" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 41857 } } peers { permanent_uuid: "f15b6ec81d284f2baa79fa54469b144b" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 35431 } attrs { promote: false } } peers { permanent_uuid: "7b496d70bc7c49da9032416864dc8ff5" member_type: NON_VOTER last_known_addr { host: "127.22.225.188" port: 34529 } attrs { promote: true } } }
I20260928 06:28:48.797652 30361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b496d70bc7c49da9032416864dc8ff5 [sys.catalog]: This master's current role is: LEARNER
W20260928 06:28:48.816288 30641 tablet.cc:2369] T 41aaf57ad53244039bfc792774f52340 P 3a7683a995ae4403b291afc021e4e0d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:28:48.891072 30504 tablet.cc:2369] T 41aaf57ad53244039bfc792774f52340 P c1813e3cd67549a1a161c3c54abc3c3b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:28:48.961823 30777 tablet.cc:2369] T 41aaf57ad53244039bfc792774f52340 P 377df16ac26d45a8b8114cc6d81d4894: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:28:49.018316 30807 raft_consensus.cc:482] T 41aaf57ad53244039bfc792774f52340 P 377df16ac26d45a8b8114cc6d81d4894 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:28:49.018463 30807 raft_consensus.cc:504] T 41aaf57ad53244039bfc792774f52340 P 377df16ac26d45a8b8114cc6d81d4894 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1813e3cd67549a1a161c3c54abc3c3b" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 40749 } } peers { permanent_uuid: "377df16ac26d45a8b8114cc6d81d4894" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 38727 } } peers { permanent_uuid: "3a7683a995ae4403b291afc021e4e0d3" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 43647 } }
I20260928 06:28:49.018857 30807 leader_election.cc:288] T 41aaf57ad53244039bfc792774f52340 P 377df16ac26d45a8b8114cc6d81d4894 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1813e3cd67549a1a161c3c54abc3c3b (127.22.225.129:40749), 3a7683a995ae4403b291afc021e4e0d3 (127.22.225.130:43647)
I20260928 06:28:49.022332 30796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:49.019215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:49629 (local address 127.22.225.129:40749)
0928 06:28:49.019258 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:49.019260 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:49.019271 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:28:49.019299 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:49.019301 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:49.019309 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:49.019381 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:28:49.019563 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:49.020021 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:49.021467 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:49.021907 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:49.022015 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:49.022023 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:49.022216 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 06:28:49.022258 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:28:49.022279 30783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:49.019027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.130:43647 (local address 127.22.225.131:54531)
0928 06:28:49.019108 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:49.019117 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:49.019147 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:49.019345 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:49.019346 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:49.019347 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:28:49.019429 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:49.019431 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:49.020345 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:49.020346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:49.021393 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:49.021394 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:49.021976 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:49.021978 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:49.022070 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:49.022074 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:28:49.022120 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 06:28:49.022191 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69,"mutex_wait_us":48}
I20260928 06:28:49.022392 30799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:49.019159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:54531 (local address 127.22.225.130:43647)
0928 06:28:49.019219 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:49.019222 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:49.019230 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:28:49.019256 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:49.019257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:49.019266 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:49.019342 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:28:49.019513 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:49.020254 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:49.021493 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:49.021885 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:49.021948 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:49.021956 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:49.022243 (+ 287us) server_negotiation.cc:299] Negotiation successful
0928 06:28:49.022292 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 06:28:49.022428 30781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:49.019027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.129:40749 (local address 127.22.225.131:49629)
0928 06:28:49.019073 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:49.019086 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:49.019129 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:49.019407 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:49.019408 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:49.019409 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:28:49.019471 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:49.019472 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:49.020110 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:49.020111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:49.021371 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:49.021374 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:49.022038 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:49.022039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:49.022152 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:49.022156 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:28:49.022208 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 06:28:49.022253 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":38}
I20260928 06:28:49.022746 30592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41aaf57ad53244039bfc792774f52340" candidate_uuid: "377df16ac26d45a8b8114cc6d81d4894" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a7683a995ae4403b291afc021e4e0d3" is_pre_election: true
I20260928 06:28:49.022742 30455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41aaf57ad53244039bfc792774f52340" candidate_uuid: "377df16ac26d45a8b8114cc6d81d4894" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1813e3cd67549a1a161c3c54abc3c3b" is_pre_election: true
I20260928 06:28:49.022943 30455 raft_consensus.cc:2454] T 41aaf57ad53244039bfc792774f52340 P c1813e3cd67549a1a161c3c54abc3c3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 377df16ac26d45a8b8114cc6d81d4894 in term 0.
I20260928 06:28:49.022949 30592 raft_consensus.cc:2454] T 41aaf57ad53244039bfc792774f52340 P 3a7683a995ae4403b291afc021e4e0d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 377df16ac26d45a8b8114cc6d81d4894 in term 0.
I20260928 06:28:49.023209 30663 leader_election.cc:302] T 41aaf57ad53244039bfc792774f52340 P 377df16ac26d45a8b8114cc6d81d4894 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 377df16ac26d45a8b8114cc6d81d4894, 3a7683a995ae4403b291afc021e4e0d3; no voters:
I20260928 06:28:49.023345 30807 raft_consensus.cc:2781] T 41aaf57ad53244039bfc792774f52340 P 377df16ac26d45a8b8114cc6d81d4894 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:28:49.023399 30807 raft_consensus.cc:482] T 41aaf57ad53244039bfc792774f52340 P 377df16ac26d45a8b8114cc6d81d4894 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:28:49.023429 30807 raft_consensus.cc:3037] T 41aaf57ad53244039bfc792774f52340 P 377df16ac26d45a8b8114cc6d81d4894 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:49.024120 30807 raft_consensus.cc:504] T 41aaf57ad53244039bfc792774f52340 P 377df16ac26d45a8b8114cc6d81d4894 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1813e3cd67549a1a161c3c54abc3c3b" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 40749 } } peers { permanent_uuid: "377df16ac26d45a8b8114cc6d81d4894" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 38727 } } peers { permanent_uuid: "3a7683a995ae4403b291afc021e4e0d3" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 43647 } }
I20260928 06:28:49.024272 30807 leader_election.cc:288] T 41aaf57ad53244039bfc792774f52340 P 377df16ac26d45a8b8114cc6d81d4894 [CANDIDATE]: Term 1 election: Requested vote from peers c1813e3cd67549a1a161c3c54abc3c3b (127.22.225.129:40749), 3a7683a995ae4403b291afc021e4e0d3 (127.22.225.130:43647)
I20260928 06:28:49.024446 30455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41aaf57ad53244039bfc792774f52340" candidate_uuid: "377df16ac26d45a8b8114cc6d81d4894" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1813e3cd67549a1a161c3c54abc3c3b"
I20260928 06:28:49.024449 30592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41aaf57ad53244039bfc792774f52340" candidate_uuid: "377df16ac26d45a8b8114cc6d81d4894" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a7683a995ae4403b291afc021e4e0d3"
I20260928 06:28:49.024544 30455 raft_consensus.cc:3037] T 41aaf57ad53244039bfc792774f52340 P c1813e3cd67549a1a161c3c54abc3c3b [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:49.024547 30592 raft_consensus.cc:3037] T 41aaf57ad53244039bfc792774f52340 P 3a7683a995ae4403b291afc021e4e0d3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:49.025487 30592 raft_consensus.cc:2454] T 41aaf57ad53244039bfc792774f52340 P 3a7683a995ae4403b291afc021e4e0d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 377df16ac26d45a8b8114cc6d81d4894 in term 1.
I20260928 06:28:49.025601 30455 raft_consensus.cc:2454] T 41aaf57ad53244039bfc792774f52340 P c1813e3cd67549a1a161c3c54abc3c3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 377df16ac26d45a8b8114cc6d81d4894 in term 1.
I20260928 06:28:49.025671 30663 leader_election.cc:302] T 41aaf57ad53244039bfc792774f52340 P 377df16ac26d45a8b8114cc6d81d4894 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 377df16ac26d45a8b8114cc6d81d4894, 3a7683a995ae4403b291afc021e4e0d3; no voters:
I20260928 06:28:49.025770 30807 raft_consensus.cc:2781] T 41aaf57ad53244039bfc792774f52340 P 377df16ac26d45a8b8114cc6d81d4894 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:28:49.025944 30807 raft_consensus.cc:686] T 41aaf57ad53244039bfc792774f52340 P 377df16ac26d45a8b8114cc6d81d4894 [term 1 LEADER]: Becoming Leader. State: Replica: 377df16ac26d45a8b8114cc6d81d4894, State: Running, Role: LEADER
I20260928 06:28:49.026041 30807 consensus_queue.cc:237] T 41aaf57ad53244039bfc792774f52340 P 377df16ac26d45a8b8114cc6d81d4894 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1813e3cd67549a1a161c3c54abc3c3b" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 40749 } } peers { permanent_uuid: "377df16ac26d45a8b8114cc6d81d4894" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 38727 } } peers { permanent_uuid: "3a7683a995ae4403b291afc021e4e0d3" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 43647 } }
I20260928 06:28:49.026922 29537 catalog_manager.cc:5495] T 41aaf57ad53244039bfc792774f52340 P 377df16ac26d45a8b8114cc6d81d4894 reported cstate change: term changed from 0 to 1, leader changed from <none> to 377df16ac26d45a8b8114cc6d81d4894 (127.22.225.131). New cstate: current_term: 1 leader_uuid: "377df16ac26d45a8b8114cc6d81d4894" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1813e3cd67549a1a161c3c54abc3c3b" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 40749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "377df16ac26d45a8b8114cc6d81d4894" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 38727 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a7683a995ae4403b291afc021e4e0d3" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 43647 } health_report { overall_health: UNKNOWN } } }
I20260928 06:28:49.027560 30098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c07c7b5cb55443299f41d25e3dc513d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0c07c7b5cb55443299f41d25e3dc513d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0c07c7b5cb55443299f41d25e3dc513d" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 41857 } } peers { permanent_uuid: "f15b6ec81d284f2baa79fa54469b144b" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 35431 } attrs { promote: false } } peers { permanent_uuid: "7b496d70bc7c49da9032416864dc8ff5" member_type: NON_VOTER last_known_addr { host: "127.22.225.188" port: 34529 } attrs { promote: true } } }
I20260928 06:28:49.027662 30098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c07c7b5cb55443299f41d25e3dc513d [sys.catalog]: This master's current role is: LEADER
I20260928 06:28:49.028270 30097 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 0c07c7b5cb55443299f41d25e3dc513d: attempting to promote NON_VOTER 7b496d70bc7c49da9032416864dc8ff5 to VOTER
I20260928 06:28:49.028393 30097 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c07c7b5cb55443299f41d25e3dc513d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0c07c7b5cb55443299f41d25e3dc513d" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 41857 } } peers { permanent_uuid: "f15b6ec81d284f2baa79fa54469b144b" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 35431 } attrs { promote: false } } peers { permanent_uuid: "7b496d70bc7c49da9032416864dc8ff5" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 34529 } attrs { promote: false } }
I20260928 06:28:49.028919 30147 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b [term 1 FOLLOWER]: Refusing update from remote peer 0c07c7b5cb55443299f41d25e3dc513d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260928 06:28:49.028978 30332 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7b496d70bc7c49da9032416864dc8ff5 [term 1 LEARNER]: Refusing update from remote peer 0c07c7b5cb55443299f41d25e3dc513d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260928 06:28:49.029127 30099 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0c07c7b5cb55443299f41d25e3dc513d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f15b6ec81d284f2baa79fa54469b144b" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 35431 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260928 06:28:49.029201 30098 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0c07c7b5cb55443299f41d25e3dc513d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b496d70bc7c49da9032416864dc8ff5" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 34529 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260928 06:28:49.029963 30097 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0c07c7b5cb55443299f41d25e3dc513d [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 7b496d70bc7c49da9032416864dc8ff5 (127.22.225.188) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0c07c7b5cb55443299f41d25e3dc513d" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 41857 } } peers { permanent_uuid: "f15b6ec81d284f2baa79fa54469b144b" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 35431 } attrs { promote: false } } peers { permanent_uuid: "7b496d70bc7c49da9032416864dc8ff5" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 34529 } attrs { promote: false } } }
I20260928 06:28:49.030164 30332 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7b496d70bc7c49da9032416864dc8ff5 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 7b496d70bc7c49da9032416864dc8ff5 (127.22.225.188) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0c07c7b5cb55443299f41d25e3dc513d" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 41857 } } peers { permanent_uuid: "f15b6ec81d284f2baa79fa54469b144b" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 35431 } attrs { promote: false } } peers { permanent_uuid: "7b496d70bc7c49da9032416864dc8ff5" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 34529 } attrs { promote: false } } }
I20260928 06:28:49.030272 30147 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 7b496d70bc7c49da9032416864dc8ff5 (127.22.225.188) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0c07c7b5cb55443299f41d25e3dc513d" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 41857 } } peers { permanent_uuid: "f15b6ec81d284f2baa79fa54469b144b" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 35431 } attrs { promote: false } } peers { permanent_uuid: "7b496d70bc7c49da9032416864dc8ff5" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 34529 } attrs { promote: false } } }
I20260928 06:28:49.030856 30097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c07c7b5cb55443299f41d25e3dc513d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0c07c7b5cb55443299f41d25e3dc513d" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0c07c7b5cb55443299f41d25e3dc513d" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 41857 } } peers { permanent_uuid: "f15b6ec81d284f2baa79fa54469b144b" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 35431 } attrs { promote: false } } peers { permanent_uuid: "7b496d70bc7c49da9032416864dc8ff5" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 34529 } attrs { promote: false } } }
I20260928 06:28:49.030931 30097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c07c7b5cb55443299f41d25e3dc513d [sys.catalog]: This master's current role is: LEADER
I20260928 06:28:49.030998 30097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c07c7b5cb55443299f41d25e3dc513d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0c07c7b5cb55443299f41d25e3dc513d" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0c07c7b5cb55443299f41d25e3dc513d" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 41857 } } peers { permanent_uuid: "f15b6ec81d284f2baa79fa54469b144b" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 35431 } attrs { promote: false } } peers { permanent_uuid: "7b496d70bc7c49da9032416864dc8ff5" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 34529 } attrs { promote: false } } }
I20260928 06:28:49.031075 30097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c07c7b5cb55443299f41d25e3dc513d [sys.catalog]: This master's current role is: LEADER
I20260928 06:28:49.031121 30097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c07c7b5cb55443299f41d25e3dc513d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0c07c7b5cb55443299f41d25e3dc513d" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0c07c7b5cb55443299f41d25e3dc513d" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 41857 } } peers { permanent_uuid: "f15b6ec81d284f2baa79fa54469b144b" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 35431 } attrs { promote: false } } peers { permanent_uuid: "7b496d70bc7c49da9032416864dc8ff5" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 34529 } attrs { promote: false } } }
I20260928 06:28:49.031177 30097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c07c7b5cb55443299f41d25e3dc513d [sys.catalog]: This master's current role is: LEADER
I20260928 06:28:49.031248 30361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b496d70bc7c49da9032416864dc8ff5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0c07c7b5cb55443299f41d25e3dc513d" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0c07c7b5cb55443299f41d25e3dc513d" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 41857 } } peers { permanent_uuid: "f15b6ec81d284f2baa79fa54469b144b" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 35431 } attrs { promote: false } } peers { permanent_uuid: "7b496d70bc7c49da9032416864dc8ff5" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 34529 } attrs { promote: false } } }
I20260928 06:28:49.031322 30361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b496d70bc7c49da9032416864dc8ff5 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:28:49.031605 30815 sys_catalog.cc:455] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0c07c7b5cb55443299f41d25e3dc513d" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0c07c7b5cb55443299f41d25e3dc513d" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 41857 } } peers { permanent_uuid: "f15b6ec81d284f2baa79fa54469b144b" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 35431 } attrs { promote: false } } peers { permanent_uuid: "7b496d70bc7c49da9032416864dc8ff5" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 34529 } attrs { promote: false } } }
I20260928 06:28:49.031729 30815 sys_catalog.cc:458] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:28:49.091503 30797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:49.089029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58982 (local address 127.22.225.131:38727)
0928 06:28:49.089085 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:49.089088 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:49.089127 (+ 39us) server_negotiation.cc:407] Connection header received
0928 06:28:49.089174 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:49.089175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:49.089198 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:49.089274 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:28:49.089409 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:49.089895 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:49.090676 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:49.090885 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:49.090956 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:49.091137 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:28:49.091251 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:28:49.091342 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:49.091380 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 06:28:49.091401 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 06:28:49.093884 30455 raft_consensus.cc:1261] T 41aaf57ad53244039bfc792774f52340 P c1813e3cd67549a1a161c3c54abc3c3b [term 1 FOLLOWER]: Refusing update from remote peer 377df16ac26d45a8b8114cc6d81d4894: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:28:49.093883 30592 raft_consensus.cc:1261] T 41aaf57ad53244039bfc792774f52340 P 3a7683a995ae4403b291afc021e4e0d3 [term 1 FOLLOWER]: Refusing update from remote peer 377df16ac26d45a8b8114cc6d81d4894: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:28:49.094295 30812 consensus_queue.cc:1035] T 41aaf57ad53244039bfc792774f52340 P 377df16ac26d45a8b8114cc6d81d4894 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1813e3cd67549a1a161c3c54abc3c3b" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 40749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:28:49.094432 30807 consensus_queue.cc:1035] T 41aaf57ad53244039bfc792774f52340 P 377df16ac26d45a8b8114cc6d81d4894 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a7683a995ae4403b291afc021e4e0d3" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:28:49.097273 30817 mvcc.cc:205] Tried to move back new op lower bound from 7334203101565153280 to 7334203101291294720. Current Snapshot: MvccSnapshot[applied={T|T < 7334203101565153280}]
I20260928 06:28:49.098798 30819 mvcc.cc:205] Tried to move back new op lower bound from 7334203101565153280 to 7334203101291294720. Current Snapshot: MvccSnapshot[applied={T|T < 7334203101565153280 or (T in {7334203101565153280})}]
I20260928 06:28:49.099021 30818 mvcc.cc:205] Tried to move back new op lower bound from 7334203101565153280 to 7334203101291294720. Current Snapshot: MvccSnapshot[applied={T|T < 7334203101565153280 or (T in {7334203101565153280})}]
I20260928 06:28:49.100494 29537 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:28:49.100618 29537 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:28:49.102669 29537 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0c07c7b5cb55443299f41d25e3dc513d: Sending DeleteTablet for 3 replicas of tablet 41aaf57ad53244039bfc792774f52340
I20260928 06:28:49.103000 30435 tablet_service.cc:1514] Processing DeleteTablet for tablet 41aaf57ad53244039bfc792774f52340 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:28:49 UTC) from {username='oryx', principal='oryx/127.22.225.190@KRBTEST.COM'} at 127.0.0.1:35048
I20260928 06:28:49.103093 30708 tablet_service.cc:1514] Processing DeleteTablet for tablet 41aaf57ad53244039bfc792774f52340 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:28:49 UTC) from {username='oryx', principal='oryx/127.22.225.190@KRBTEST.COM'} at 127.0.0.1:58974
I20260928 06:28:49.103298 30828 tablet_replica.cc:330] T 41aaf57ad53244039bfc792774f52340 P 377df16ac26d45a8b8114cc6d81d4894: stopping tablet replica
I20260928 06:28:49.103400 30828 raft_consensus.cc:2229] T 41aaf57ad53244039bfc792774f52340 P 377df16ac26d45a8b8114cc6d81d4894 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:28:49.103408 30572 tablet_service.cc:1514] Processing DeleteTablet for tablet 41aaf57ad53244039bfc792774f52340 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:28:49 UTC) from {username='oryx', principal='oryx/127.22.225.190@KRBTEST.COM'} at 127.0.0.1:60230
I20260928 06:28:49.103500 30828 raft_consensus.cc:2258] T 41aaf57ad53244039bfc792774f52340 P 377df16ac26d45a8b8114cc6d81d4894 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:28:49.103672 30829 tablet_replica.cc:330] T 41aaf57ad53244039bfc792774f52340 P 3a7683a995ae4403b291afc021e4e0d3: stopping tablet replica
I20260928 06:28:49.103869 30829 raft_consensus.cc:2229] T 41aaf57ad53244039bfc792774f52340 P 3a7683a995ae4403b291afc021e4e0d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:28:49.104046 30829 raft_consensus.cc:2258] T 41aaf57ad53244039bfc792774f52340 P 3a7683a995ae4403b291afc021e4e0d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:28:49.104246 30828 ts_tablet_manager.cc:1905] T 41aaf57ad53244039bfc792774f52340 P 377df16ac26d45a8b8114cc6d81d4894: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:28:49.104624 30829 ts_tablet_manager.cc:1905] T 41aaf57ad53244039bfc792774f52340 P 3a7683a995ae4403b291afc021e4e0d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:28:49.104785 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 29506
I20260928 06:28:49.106343 30828 ts_tablet_manager.cc:1918] T 41aaf57ad53244039bfc792774f52340 P 377df16ac26d45a8b8114cc6d81d4894: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:28:49.106357 30829 ts_tablet_manager.cc:1918] T 41aaf57ad53244039bfc792774f52340 P 3a7683a995ae4403b291afc021e4e0d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:28:49.106426 30828 log.cc:1196] T 41aaf57ad53244039bfc792774f52340 P 377df16ac26d45a8b8114cc6d81d4894: Deleting WAL directory at /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-2/wal/wals/41aaf57ad53244039bfc792774f52340
I20260928 06:28:49.106547 30829 log.cc:1196] T 41aaf57ad53244039bfc792774f52340 P 3a7683a995ae4403b291afc021e4e0d3: Deleting WAL directory at /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-1/wal/wals/41aaf57ad53244039bfc792774f52340
I20260928 06:28:49.106720 30828 ts_tablet_manager.cc:1939] T 41aaf57ad53244039bfc792774f52340 P 377df16ac26d45a8b8114cc6d81d4894: Deleting consensus metadata
I20260928 06:28:49.107017 30829 ts_tablet_manager.cc:1939] T 41aaf57ad53244039bfc792774f52340 P 3a7683a995ae4403b291afc021e4e0d3: Deleting consensus metadata
W20260928 06:28:49.113525 30526 connection.cc:535] server connection from 127.0.0.1:60230 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 06:28:49.113754 30664 connection.cc:535] server connection from 127.0.0.1:58974 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 06:28:49.114859 30827 tablet_replica.cc:330] T 41aaf57ad53244039bfc792774f52340 P c1813e3cd67549a1a161c3c54abc3c3b: stopping tablet replica
I20260928 06:28:49.115010 30827 raft_consensus.cc:2229] T 41aaf57ad53244039bfc792774f52340 P c1813e3cd67549a1a161c3c54abc3c3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:28:49.115170 30827 raft_consensus.cc:2258] T 41aaf57ad53244039bfc792774f52340 P c1813e3cd67549a1a161c3c54abc3c3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:28:49.115937 30827 ts_tablet_manager.cc:1905] T 41aaf57ad53244039bfc792774f52340 P c1813e3cd67549a1a161c3c54abc3c3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:28:49.118583 30827 ts_tablet_manager.cc:1918] T 41aaf57ad53244039bfc792774f52340 P c1813e3cd67549a1a161c3c54abc3c3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:28:49.118683 30827 log.cc:1196] T 41aaf57ad53244039bfc792774f52340 P c1813e3cd67549a1a161c3c54abc3c3b: Deleting WAL directory at /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-0/wal/wals/41aaf57ad53244039bfc792774f52340
I20260928 06:28:49.118988 30827 ts_tablet_manager.cc:1939] T 41aaf57ad53244039bfc792774f52340 P c1813e3cd67549a1a161c3c54abc3c3b: Deleting consensus metadata
W20260928 06:28:49.119549 30391 connection.cc:441] server connection from 127.0.0.1:35048 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35048 (request call id 1) could send its response
I20260928 06:28:49.124022 30505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:49.103798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42220 (local address 127.22.225.188:34529)
0928 06:28:49.103851 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:49.103854 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:49.115420 (+ 11566us) server_negotiation.cc:407] Connection header received
0928 06:28:49.115464 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:49.115465 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:49.115478 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:49.115537 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:49.116477 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:49.117159 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:49.118218 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:49.118525 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:49.118574 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:49.120626 (+ 2052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:49.120627 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:49.120627 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:49.120639 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:49.122743 (+ 2104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:49.123109 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:49.123110 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:49.123110 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:49.123172 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:49.123391 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:49.123394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:49.123394 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:49.123529 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:49.123625 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:49.123823 (+ 198us) server_negotiation.cc:299] Negotiation successful
0928 06:28:49.123899 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 06:28:49.126858 30188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:49.115569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44640 (local address 127.22.225.189:35431)
0928 06:28:49.115614 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:49.115616 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:49.116783 (+ 1167us) server_negotiation.cc:407] Connection header received
0928 06:28:49.116881 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:49.116882 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:49.116902 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:49.116933 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:49.117676 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:49.118433 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:49.120037 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:49.120332 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:49.120371 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:49.121855 (+ 1484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:49.121857 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:49.121857 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:49.121865 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:49.125529 (+ 3664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:49.125992 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:49.125993 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:49.125994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:49.126055 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:49.126276 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:49.126277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:49.126277 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:49.126399 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:49.126473 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:49.126671 (+ 198us) server_negotiation.cc:299] Negotiation successful
0928 06:28:49.126749 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:28:49.128289 30789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:49.117290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42222 (local address 127.22.225.188:34529)
0928 06:28:49.117330 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:49.117333 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:49.119736 (+ 2403us) server_negotiation.cc:407] Connection header received
0928 06:28:49.119840 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:49.119841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:49.119858 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:49.119887 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:49.120453 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:49.121645 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:49.122697 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:49.123030 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:49.123070 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:49.125215 (+ 2145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:49.125216 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:49.125216 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:49.125222 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:49.127093 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:49.127520 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:49.127521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:49.127521 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:49.127570 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:49.127752 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:49.127753 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:49.127753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:49.127864 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:49.127941 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:49.128132 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 06:28:49.128209 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:28:50.098841 30838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:50.098295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:41857 (local address 127.22.225.130:41753)
0928 06:28:50.098551 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:50.098777 (+ 226us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":87,"threads_started":1}
W20260928 06:28:50.099172 30639 heartbeater.cc:644] Failed to heartbeat to 127.22.225.190:41857 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.22.225.190:41857: connect: Connection refused (error 111)
I20260928 06:28:50.099373 30839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:50.098296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:41857 (local address 127.22.225.131:41661)
0928 06:28:50.099150 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:50.099217 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":107,"threads_started":1}
W20260928 06:28:50.099560 30775 heartbeater.cc:644] Failed to heartbeat to 127.22.225.190:41857 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.22.225.190:41857: connect: Connection refused (error 111)
I20260928 06:28:50.100940 30840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:50.100703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:41857 (local address 127.22.225.129:45809)
0928 06:28:50.100828 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:50.100895 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":73,"threads_started":1}
W20260928 06:28:50.101135 30502 heartbeater.cc:644] Failed to heartbeat to 127.22.225.190:41857 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.22.225.190:41857: connect: Connection refused (error 111)
I20260928 06:28:50.602770 30841 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b496d70bc7c49da9032416864dc8ff5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0c07c7b5cb55443299f41d25e3dc513d)
I20260928 06:28:50.602949 30841 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b496d70bc7c49da9032416864dc8ff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0c07c7b5cb55443299f41d25e3dc513d" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 41857 } } peers { permanent_uuid: "f15b6ec81d284f2baa79fa54469b144b" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 35431 } attrs { promote: false } } peers { permanent_uuid: "7b496d70bc7c49da9032416864dc8ff5" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 34529 } attrs { promote: false } }
I20260928 06:28:50.603281 30841 leader_election.cc:288] T 00000000000000000000000000000000 P 7b496d70bc7c49da9032416864dc8ff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0c07c7b5cb55443299f41d25e3dc513d (127.22.225.190:41857), f15b6ec81d284f2baa79fa54469b144b (127.22.225.189:35431)
I20260928 06:28:50.603559 30842 raft_consensus.cc:482] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0c07c7b5cb55443299f41d25e3dc513d)
I20260928 06:28:50.603675 30844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:50.603425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:41857 (local address 127.0.0.1:54188)
0928 06:28:50.603561 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:50.603633 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":48,"threads_started":1}
I20260928 06:28:50.603698 30842 raft_consensus.cc:504] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0c07c7b5cb55443299f41d25e3dc513d" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 41857 } } peers { permanent_uuid: "f15b6ec81d284f2baa79fa54469b144b" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 35431 } attrs { promote: false } } peers { permanent_uuid: "7b496d70bc7c49da9032416864dc8ff5" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 34529 } attrs { promote: false } }
W20260928 06:28:50.603780 30309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.225.190:41857: connect: Connection refused (error 111)
I20260928 06:28:50.603932 30842 leader_election.cc:288] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0c07c7b5cb55443299f41d25e3dc513d (127.22.225.190:41857), 7b496d70bc7c49da9032416864dc8ff5 (127.22.225.188:34529)
I20260928 06:28:50.604208 30847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:50.604022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:41857 (local address 127.0.0.1:54192)
0928 06:28:50.604106 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:50.604169 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71,"thread_start_us":50,"threads_started":1}
W20260928 06:28:50.604341 30124 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.225.190:41857: connect: Connection refused (error 111)
I20260928 06:28:50.604782 30848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:50.604683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:41857 (local address 127.0.0.1:54210)
0928 06:28:50.604722 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:50.604752 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 06:28:50.604854 30124 leader_election.cc:334] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0c07c7b5cb55443299f41d25e3dc513d (127.22.225.190:41857): Network error: Client connection negotiation failed: client connection to 127.22.225.190:41857: connect: Connection refused (error 111)
I20260928 06:28:50.604925 30844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:50.604204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:41857 (local address 127.0.0.1:54206)
0928 06:28:50.604249 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:50.604561 (+ 312us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36,"thread_start_us":78,"threads_started":1}
W20260928 06:28:50.604996 30309 leader_election.cc:334] T 00000000000000000000000000000000 P 7b496d70bc7c49da9032416864dc8ff5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0c07c7b5cb55443299f41d25e3dc513d (127.22.225.190:41857): Network error: Client connection negotiation failed: client connection to 127.22.225.190:41857: connect: Connection refused (error 111)
I20260928 06:28:50.607730 30847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:50.604166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.188:34529 (local address 127.0.0.1:42226)
0928 06:28:50.604286 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:50.604296 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:50.604366 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:50.604871 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:50.604872 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:50.604875 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:28:50.604916 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:50.604918 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:50.605731 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:50.605732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:50.606967 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:50.606970 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:50.607483 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:50.607484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:50.607555 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:50.607560 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:28:50.607608 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 06:28:50.607652 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":41,"threads_started":1}
I20260928 06:28:50.607734 30850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:50.604408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42226 (local address 127.22.225.188:34529)
0928 06:28:50.604714 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:50.604716 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:50.604724 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:28:50.604745 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:50.604745 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:50.604755 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:50.604830 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:28:50.604982 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:50.605656 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:50.607041 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:50.607390 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:50.607483 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:50.607492 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:50.607630 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 06:28:50.607670 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":67,"threads_started":1}
I20260928 06:28:50.607944 30332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f15b6ec81d284f2baa79fa54469b144b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7b496d70bc7c49da9032416864dc8ff5" is_pre_election: true
I20260928 06:28:50.608037 30332 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7b496d70bc7c49da9032416864dc8ff5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f15b6ec81d284f2baa79fa54469b144b in term 1.
I20260928 06:28:50.608181 30846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:50.603879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44644 (local address 127.22.225.189:35431)
0928 06:28:50.604036 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:50.604038 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:50.604141 (+ 103us) server_negotiation.cc:407] Connection header received
0928 06:28:50.604255 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:50.604256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:50.604268 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:50.604309 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:28:50.604665 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:50.605574 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:50.606990 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:50.607495 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:50.607574 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:50.607581 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:50.607710 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 06:28:50.607751 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":53,"threads_started":1}
I20260928 06:28:50.608204 30124 leader_election.cc:302] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7b496d70bc7c49da9032416864dc8ff5, f15b6ec81d284f2baa79fa54469b144b; no voters: 0c07c7b5cb55443299f41d25e3dc513d
I20260928 06:28:50.608312 30842 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 06:28:50.608345 30842 raft_consensus.cc:482] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0c07c7b5cb55443299f41d25e3dc513d)
I20260928 06:28:50.608362 30842 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b [term 1 FOLLOWER]: Advancing to term 2
I20260928 06:28:50.608425 30843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:50.603461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.189:35431 (local address 127.0.0.1:44644)
0928 06:28:50.604073 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:50.604084 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:50.604170 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:50.604481 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:50.604482 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:50.604485 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:28:50.604545 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:50.604548 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:50.605668 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:50.605669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:50.606890 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:50.606893 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:50.607572 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:50.607572 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:50.607629 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:50.607632 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:28:50.608312 (+ 680us) client_negotiation.cc:240] Negotiation successful
0928 06:28:50.608355 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":55,"threads_started":1}
I20260928 06:28:50.608649 30147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7b496d70bc7c49da9032416864dc8ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f15b6ec81d284f2baa79fa54469b144b" is_pre_election: true
I20260928 06:28:50.609300 30842 raft_consensus.cc:504] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0c07c7b5cb55443299f41d25e3dc513d" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 41857 } } peers { permanent_uuid: "f15b6ec81d284f2baa79fa54469b144b" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 35431 } attrs { promote: false } } peers { permanent_uuid: "7b496d70bc7c49da9032416864dc8ff5" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 34529 } attrs { promote: false } }
I20260928 06:28:50.609437 30147 raft_consensus.cc:2379] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7b496d70bc7c49da9032416864dc8ff5 in current term 2: Already voted for candidate f15b6ec81d284f2baa79fa54469b144b in this term.
I20260928 06:28:50.609440 30842 leader_election.cc:288] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b [CANDIDATE]: Term 2 election: Requested vote from peers 0c07c7b5cb55443299f41d25e3dc513d (127.22.225.190:41857), 7b496d70bc7c49da9032416864dc8ff5 (127.22.225.188:34529)
I20260928 06:28:50.609690 30308 leader_election.cc:302] T 00000000000000000000000000000000 P 7b496d70bc7c49da9032416864dc8ff5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7b496d70bc7c49da9032416864dc8ff5; no voters: 0c07c7b5cb55443299f41d25e3dc513d, f15b6ec81d284f2baa79fa54469b144b
I20260928 06:28:50.609720 30847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:50.609635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:41857 (local address 127.0.0.1:54214)
0928 06:28:50.609674 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:50.609700 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260928 06:28:50.609872 30841 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b496d70bc7c49da9032416864dc8ff5 [term 1 FOLLOWER]: Advancing to term 2
I20260928 06:28:50.609977 30332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f15b6ec81d284f2baa79fa54469b144b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7b496d70bc7c49da9032416864dc8ff5"
I20260928 06:28:50.610229 30847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:50.609954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:41857 (local address 127.0.0.1:54220)
0928 06:28:50.610138 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:50.610208 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172}
W20260928 06:28:50.610280 30124 leader_election.cc:334] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 0c07c7b5cb55443299f41d25e3dc513d (127.22.225.190:41857): Network error: Client connection negotiation failed: client connection to 127.22.225.190:41857: connect: Connection refused (error 111)
I20260928 06:28:50.610637 30841 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 7b496d70bc7c49da9032416864dc8ff5 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:28:50.611469 30332 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7b496d70bc7c49da9032416864dc8ff5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f15b6ec81d284f2baa79fa54469b144b in term 2.
I20260928 06:28:50.611696 30124 leader_election.cc:302] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7b496d70bc7c49da9032416864dc8ff5, f15b6ec81d284f2baa79fa54469b144b; no voters: 0c07c7b5cb55443299f41d25e3dc513d
I20260928 06:28:50.611814 30842 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b [term 2 FOLLOWER]: Leader election won for term 2
I20260928 06:28:50.612007 30842 raft_consensus.cc:686] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b [term 2 LEADER]: Becoming Leader. State: Replica: f15b6ec81d284f2baa79fa54469b144b, State: Running, Role: LEADER
I20260928 06:28:50.612088 30842 consensus_queue.cc:237] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0c07c7b5cb55443299f41d25e3dc513d" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 41857 } } peers { permanent_uuid: "f15b6ec81d284f2baa79fa54469b144b" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 35431 } attrs { promote: false } } peers { permanent_uuid: "7b496d70bc7c49da9032416864dc8ff5" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 34529 } attrs { promote: false } }
I20260928 06:28:50.612442 30852 sys_catalog.cc:455] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f15b6ec81d284f2baa79fa54469b144b. Latest consensus state: current_term: 2 leader_uuid: "f15b6ec81d284f2baa79fa54469b144b" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0c07c7b5cb55443299f41d25e3dc513d" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 41857 } } peers { permanent_uuid: "f15b6ec81d284f2baa79fa54469b144b" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 35431 } attrs { promote: false } } peers { permanent_uuid: "7b496d70bc7c49da9032416864dc8ff5" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 34529 } attrs { promote: false } } }
I20260928 06:28:50.612530 30852 sys_catalog.cc:458] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b [sys.catalog]: This master's current role is: LEADER
I20260928 06:28:50.612720 30854 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:28:50.613112 30854 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:28:50.613204 30854 catalog_manager.cc:1259] Loaded cluster ID: 3fb00626f39f44199dbf917853eb1818
I20260928 06:28:50.613234 30854 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:28:50.613387 30854 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:28:50.613481 30854 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b: Loaded TSK: 0
I20260928 06:28:50.613622 30854 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:28:50.668084 30137 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44640:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:28:50.670223 30137 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:28:50.671247 30847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:50.671132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:41857 (local address 127.0.0.1:54246)
0928 06:28:50.671185 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:50.671215 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 06:28:50.671378 30332 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7b496d70bc7c49da9032416864dc8ff5 [term 2 FOLLOWER]: Refusing update from remote peer f15b6ec81d284f2baa79fa54469b144b: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260928 06:28:50.671650 30852 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b496d70bc7c49da9032416864dc8ff5" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 34529 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260928 06:28:50.671777 30847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:50.671676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:41857 (local address 127.0.0.1:54250)
0928 06:28:50.671721 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:50.671746 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 06:28:50.671947 30124 consensus_peers.cc:487] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b -> Peer 0c07c7b5cb55443299f41d25e3dc513d (127.22.225.190:41857): Couldn't send request to peer 0c07c7b5cb55443299f41d25e3dc513d. Status: Network error: Client connection negotiation failed: client connection to 127.22.225.190:41857: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 06:28:50.672410 30841 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b496d70bc7c49da9032416864dc8ff5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f15b6ec81d284f2baa79fa54469b144b. Latest consensus state: current_term: 2 leader_uuid: "f15b6ec81d284f2baa79fa54469b144b" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0c07c7b5cb55443299f41d25e3dc513d" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 41857 } } peers { permanent_uuid: "f15b6ec81d284f2baa79fa54469b144b" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 35431 } attrs { promote: false } } peers { permanent_uuid: "7b496d70bc7c49da9032416864dc8ff5" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 34529 } attrs { promote: false } } }
I20260928 06:28:50.672503 30841 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b496d70bc7c49da9032416864dc8ff5 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:28:50.672997 30852 sys_catalog.cc:455] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "f15b6ec81d284f2baa79fa54469b144b" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0c07c7b5cb55443299f41d25e3dc513d" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 41857 } } peers { permanent_uuid: "f15b6ec81d284f2baa79fa54469b144b" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 35431 } attrs { promote: false } } peers { permanent_uuid: "7b496d70bc7c49da9032416864dc8ff5" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 34529 } attrs { promote: false } } }
I20260928 06:28:50.673101 30852 sys_catalog.cc:458] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b [sys.catalog]: This master's current role is: LEADER
I20260928 06:28:50.673189 30856 mvcc.cc:205] Tried to move back new op lower bound from 7334203108027580416 to 7334203107787755520. Current Snapshot: MvccSnapshot[applied={T|T < 7334203108027580416}]
I20260928 06:28:50.673280 30841 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b496d70bc7c49da9032416864dc8ff5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "f15b6ec81d284f2baa79fa54469b144b" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0c07c7b5cb55443299f41d25e3dc513d" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 41857 } } peers { permanent_uuid: "f15b6ec81d284f2baa79fa54469b144b" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 35431 } attrs { promote: false } } peers { permanent_uuid: "7b496d70bc7c49da9032416864dc8ff5" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 34529 } attrs { promote: false } } }
I20260928 06:28:50.673352 30841 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b496d70bc7c49da9032416864dc8ff5 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:28:50.678892 30848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:50.675017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.130:43647 (local address 127.0.0.1:60236)
0928 06:28:50.675200 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:50.675209 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:50.675246 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:50.675605 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:50.675606 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:50.675609 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:28:50.675657 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:50.675659 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:50.676714 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:50.676714 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:50.677894 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:50.677896 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:50.678634 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:50.678635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:50.678704 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:50.678708 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:28:50.678737 (+ 29us) client_negotiation.cc:240] Negotiation successful
0928 06:28:50.678802 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":42}
I20260928 06:28:50.679044 30862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:50.675126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35054 (local address 127.22.225.129:40749)
0928 06:28:50.675573 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:50.675575 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:50.675582 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:28:50.675607 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:50.675607 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:50.675617 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:50.675675 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:28:50.675807 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:50.676499 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:50.678027 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:50.678515 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:50.678849 (+ 334us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:50.678858 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:50.678877 (+ 19us) server_negotiation.cc:299] Negotiation successful
0928 06:28:50.678911 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":58,"threads_started":1}
I20260928 06:28:50.679066 30863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:50.675140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58996 (local address 127.22.225.131:38727)
0928 06:28:50.675380 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:50.675383 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:50.675391 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:28:50.675419 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:50.675420 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:50.675430 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:50.675494 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:28:50.675723 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:50.676521 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:50.677987 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:50.678525 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:50.678600 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:50.678611 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:50.678927 (+ 316us) server_negotiation.cc:299] Negotiation successful
0928 06:28:50.678973 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":66,"threads_started":1}
I20260928 06:28:50.679306 30847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:50.674999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.129:40749 (local address 127.0.0.1:35054)
0928 06:28:50.675087 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:50.675097 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:50.675138 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:50.675713 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:50.675713 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:50.675716 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:28:50.675747 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:50.675748 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:50.676625 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:50.676626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:50.677688 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:50.677691 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:50.678609 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:50.678610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:50.678668 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:50.678673 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:28:50.678700 (+ 27us) client_negotiation.cc:240] Negotiation successful
0928 06:28:50.678744 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71,"mutex_wait_us":42}
I20260928 06:28:50.679566 30435 tablet_service.cc:1467] Processing CreateTablet for tablet 2932c651a3e047d484822aae9f61edb4 (DEFAULT_TABLE table=test-table [id=c0a616827b6c4a77827b1c41ede05886]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:28:50.679596 30864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:50.675239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60236 (local address 127.22.225.130:43647)
0928 06:28:50.675347 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:50.675350 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:50.675358 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:28:50.675388 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:50.675389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:50.675402 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:50.675829 (+ 427us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:28:50.675868 (+ 39us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:50.676630 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:50.677988 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:50.678549 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:50.679442 (+ 893us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:50.679454 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:50.679475 (+ 21us) server_negotiation.cc:299] Negotiation successful
0928 06:28:50.679518 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":68,"threads_started":1}
I20260928 06:28:50.678900 30861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:50.675078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.131:38727 (local address 127.0.0.1:58996)
0928 06:28:50.675275 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:50.675285 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:50.675329 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:50.675498 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:50.675498 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:50.675501 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:28:50.675552 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:50.675554 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:50.676641 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:50.676642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:50.677893 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:50.677896 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:50.678637 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:50.678637 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:50.678706 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:50.678708 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:28:50.678762 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 06:28:50.678813 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":41,"thread_start_us":66,"threads_started":1}
I20260928 06:28:50.679736 30435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2932c651a3e047d484822aae9f61edb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:50.679809 30572 tablet_service.cc:1467] Processing CreateTablet for tablet 2932c651a3e047d484822aae9f61edb4 (DEFAULT_TABLE table=test-table [id=c0a616827b6c4a77827b1c41ede05886]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:28:50.679908 30572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2932c651a3e047d484822aae9f61edb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:50.680156 30708 tablet_service.cc:1467] Processing CreateTablet for tablet 2932c651a3e047d484822aae9f61edb4 (DEFAULT_TABLE table=test-table [id=c0a616827b6c4a77827b1c41ede05886]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:28:50.680265 30708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2932c651a3e047d484822aae9f61edb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:50.681178 30865 tablet_bootstrap.cc:492] T 2932c651a3e047d484822aae9f61edb4 P 3a7683a995ae4403b291afc021e4e0d3: Bootstrap starting.
I20260928 06:28:50.681309 30866 tablet_bootstrap.cc:492] T 2932c651a3e047d484822aae9f61edb4 P c1813e3cd67549a1a161c3c54abc3c3b: Bootstrap starting.
I20260928 06:28:50.681516 30865 tablet_bootstrap.cc:654] T 2932c651a3e047d484822aae9f61edb4 P 3a7683a995ae4403b291afc021e4e0d3: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:50.681619 30866 tablet_bootstrap.cc:654] T 2932c651a3e047d484822aae9f61edb4 P c1813e3cd67549a1a161c3c54abc3c3b: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:50.682128 30865 tablet_bootstrap.cc:492] T 2932c651a3e047d484822aae9f61edb4 P 3a7683a995ae4403b291afc021e4e0d3: No bootstrap required, opened a new log
I20260928 06:28:50.682194 30865 ts_tablet_manager.cc:1397] T 2932c651a3e047d484822aae9f61edb4 P 3a7683a995ae4403b291afc021e4e0d3: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:28:50.682452 30865 raft_consensus.cc:348] T 2932c651a3e047d484822aae9f61edb4 P 3a7683a995ae4403b291afc021e4e0d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1813e3cd67549a1a161c3c54abc3c3b" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 40749 } } peers { permanent_uuid: "3a7683a995ae4403b291afc021e4e0d3" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 43647 } } peers { permanent_uuid: "377df16ac26d45a8b8114cc6d81d4894" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 38727 } }
I20260928 06:28:50.682561 30865 raft_consensus.cc:374] T 2932c651a3e047d484822aae9f61edb4 P 3a7683a995ae4403b291afc021e4e0d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:50.682590 30865 raft_consensus.cc:729] T 2932c651a3e047d484822aae9f61edb4 P 3a7683a995ae4403b291afc021e4e0d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a7683a995ae4403b291afc021e4e0d3, State: Initialized, Role: FOLLOWER
I20260928 06:28:50.682586 30866 tablet_bootstrap.cc:492] T 2932c651a3e047d484822aae9f61edb4 P c1813e3cd67549a1a161c3c54abc3c3b: No bootstrap required, opened a new log
I20260928 06:28:50.682636 30866 ts_tablet_manager.cc:1397] T 2932c651a3e047d484822aae9f61edb4 P c1813e3cd67549a1a161c3c54abc3c3b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:28:50.682651 30865 consensus_queue.cc:260] T 2932c651a3e047d484822aae9f61edb4 P 3a7683a995ae4403b291afc021e4e0d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1813e3cd67549a1a161c3c54abc3c3b" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 40749 } } peers { permanent_uuid: "3a7683a995ae4403b291afc021e4e0d3" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 43647 } } peers { permanent_uuid: "377df16ac26d45a8b8114cc6d81d4894" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 38727 } }
I20260928 06:28:50.682794 30865 ts_tablet_manager.cc:1428] T 2932c651a3e047d484822aae9f61edb4 P 3a7683a995ae4403b291afc021e4e0d3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:28:50.682794 30866 raft_consensus.cc:348] T 2932c651a3e047d484822aae9f61edb4 P c1813e3cd67549a1a161c3c54abc3c3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1813e3cd67549a1a161c3c54abc3c3b" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 40749 } } peers { permanent_uuid: "3a7683a995ae4403b291afc021e4e0d3" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 43647 } } peers { permanent_uuid: "377df16ac26d45a8b8114cc6d81d4894" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 38727 } }
I20260928 06:28:50.682883 30866 raft_consensus.cc:374] T 2932c651a3e047d484822aae9f61edb4 P c1813e3cd67549a1a161c3c54abc3c3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:50.682906 30866 raft_consensus.cc:729] T 2932c651a3e047d484822aae9f61edb4 P c1813e3cd67549a1a161c3c54abc3c3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1813e3cd67549a1a161c3c54abc3c3b, State: Initialized, Role: FOLLOWER
I20260928 06:28:50.682942 30866 consensus_queue.cc:260] T 2932c651a3e047d484822aae9f61edb4 P c1813e3cd67549a1a161c3c54abc3c3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1813e3cd67549a1a161c3c54abc3c3b" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 40749 } } peers { permanent_uuid: "3a7683a995ae4403b291afc021e4e0d3" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 43647 } } peers { permanent_uuid: "377df16ac26d45a8b8114cc6d81d4894" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 38727 } }
I20260928 06:28:50.683087 30866 ts_tablet_manager.cc:1428] T 2932c651a3e047d484822aae9f61edb4 P c1813e3cd67549a1a161c3c54abc3c3b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 06:28:50.682801 30869 tablet_bootstrap.cc:492] T 2932c651a3e047d484822aae9f61edb4 P 377df16ac26d45a8b8114cc6d81d4894: Bootstrap starting.
I20260928 06:28:50.683844 30869 tablet_bootstrap.cc:654] T 2932c651a3e047d484822aae9f61edb4 P 377df16ac26d45a8b8114cc6d81d4894: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:50.684194 30872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:50.683015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:41857 (local address 127.22.225.130:39759)
0928 06:28:50.684119 (+ 1104us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:50.684162 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1087,"thread_start_us":88,"threads_started":1}
I20260928 06:28:50.684194 30503 heartbeater.cc:503] Master 127.22.225.189:35431 was elected leader, sending a full tablet report...
I20260928 06:28:50.684623 30872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:50.684490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:41857 (local address 127.22.225.130:48519)
0928 06:28:50.684544 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:50.684570 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 06:28:50.684664 30640 heartbeater.cc:503] Master 127.22.225.189:35431 was elected leader, sending a full tablet report...
I20260928 06:28:50.685321 30873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:50.683324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:41857 (local address 127.22.225.129:47325)
0928 06:28:50.685244 (+ 1920us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:50.685282 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1899,"thread_start_us":74,"threads_started":1}
I20260928 06:28:50.685331 30869 tablet_bootstrap.cc:492] T 2932c651a3e047d484822aae9f61edb4 P 377df16ac26d45a8b8114cc6d81d4894: No bootstrap required, opened a new log
I20260928 06:28:50.685396 30869 ts_tablet_manager.cc:1397] T 2932c651a3e047d484822aae9f61edb4 P 377df16ac26d45a8b8114cc6d81d4894: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 06:28:50.685519 30869 raft_consensus.cc:348] T 2932c651a3e047d484822aae9f61edb4 P 377df16ac26d45a8b8114cc6d81d4894 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1813e3cd67549a1a161c3c54abc3c3b" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 40749 } } peers { permanent_uuid: "3a7683a995ae4403b291afc021e4e0d3" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 43647 } } peers { permanent_uuid: "377df16ac26d45a8b8114cc6d81d4894" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 38727 } }
I20260928 06:28:50.685602 30869 raft_consensus.cc:374] T 2932c651a3e047d484822aae9f61edb4 P 377df16ac26d45a8b8114cc6d81d4894 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:50.685631 30869 raft_consensus.cc:729] T 2932c651a3e047d484822aae9f61edb4 P 377df16ac26d45a8b8114cc6d81d4894 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 377df16ac26d45a8b8114cc6d81d4894, State: Initialized, Role: FOLLOWER
I20260928 06:28:50.685676 30869 consensus_queue.cc:260] T 2932c651a3e047d484822aae9f61edb4 P 377df16ac26d45a8b8114cc6d81d4894 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1813e3cd67549a1a161c3c54abc3c3b" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 40749 } } peers { permanent_uuid: "3a7683a995ae4403b291afc021e4e0d3" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 43647 } } peers { permanent_uuid: "377df16ac26d45a8b8114cc6d81d4894" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 38727 } }
I20260928 06:28:50.685704 30875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:50.685524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:41857 (local address 127.22.225.129:46199)
0928 06:28:50.685636 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:50.685668 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":76,"threads_started":1}
I20260928 06:28:50.685819 30869 ts_tablet_manager.cc:1428] T 2932c651a3e047d484822aae9f61edb4 P 377df16ac26d45a8b8114cc6d81d4894: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 06:28:50.686422 30776 heartbeater.cc:503] Master 127.22.225.189:35431 was elected leader, sending a full tablet report...
I20260928 06:28:50.686504 30877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:50.686125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:41857 (local address 127.22.225.131:58721)
0928 06:28:50.686431 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:50.686464 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":72,"threads_started":1}
I20260928 06:28:50.686822 30877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:50.686699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:41857 (local address 127.22.225.131:52403)
0928 06:28:50.686752 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:50.686801 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 06:28:50.997941 30870 raft_consensus.cc:482] T 2932c651a3e047d484822aae9f61edb4 P 3a7683a995ae4403b291afc021e4e0d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:28:50.998061 30870 raft_consensus.cc:504] T 2932c651a3e047d484822aae9f61edb4 P 3a7683a995ae4403b291afc021e4e0d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1813e3cd67549a1a161c3c54abc3c3b" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 40749 } } peers { permanent_uuid: "3a7683a995ae4403b291afc021e4e0d3" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 43647 } } peers { permanent_uuid: "377df16ac26d45a8b8114cc6d81d4894" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 38727 } }
I20260928 06:28:50.998428 30870 leader_election.cc:288] T 2932c651a3e047d484822aae9f61edb4 P 3a7683a995ae4403b291afc021e4e0d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1813e3cd67549a1a161c3c54abc3c3b (127.22.225.129:40749), 377df16ac26d45a8b8114cc6d81d4894 (127.22.225.131:38727)
I20260928 06:28:51.001557 30872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:50.998623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.131:38727 (local address 127.22.225.130:50507)
0928 06:28:50.998691 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:50.998704 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:50.998873 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:50.999126 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:50.999127 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:50.999129 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:28:50.999180 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:50.999181 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:50.999808 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:50.999809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:51.000717 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:51.000720 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:51.001260 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:51.001261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:51.001336 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:51.001340 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:28:51.001421 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 06:28:51.001467 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":3}
I20260928 06:28:51.001549 30863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:50.998693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:50507 (local address 127.22.225.131:38727)
0928 06:28:50.998740 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:50.998742 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:50.998855 (+ 113us) server_negotiation.cc:407] Connection header received
0928 06:28:50.998944 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:50.998945 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:50.998956 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:50.999009 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:28:50.999257 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:50.999723 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:51.000803 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:51.001162 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:51.001258 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:51.001269 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:51.001430 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 06:28:51.001470 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 06:28:51.001547 30862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:50.998675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:39309 (local address 127.22.225.129:40749)
0928 06:28:50.998728 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:50.998730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:50.998955 (+ 225us) server_negotiation.cc:407] Connection header received
0928 06:28:50.999010 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:50.999011 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:50.999020 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:50.999079 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:28:50.999228 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:50.999708 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:51.000804 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:51.001165 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:51.001259 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:51.001269 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:51.001434 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 06:28:51.001471 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 06:28:51.001575 30878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:50.998624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.129:40749 (local address 127.22.225.130:39309)
0928 06:28:50.998898 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:50.998908 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:50.998963 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:50.999091 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:50.999091 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:50.999093 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:28:50.999157 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:50.999159 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:50.999789 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:50.999790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:51.000722 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:51.000724 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:51.001276 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:51.001277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:51.001350 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:51.001353 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:28:51.001425 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 06:28:51.001466 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":124,"thread_start_us":136,"threads_started":1}
I20260928 06:28:51.001780 30728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2932c651a3e047d484822aae9f61edb4" candidate_uuid: "3a7683a995ae4403b291afc021e4e0d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "377df16ac26d45a8b8114cc6d81d4894" is_pre_election: true
I20260928 06:28:51.001919 30728 raft_consensus.cc:2454] T 2932c651a3e047d484822aae9f61edb4 P 377df16ac26d45a8b8114cc6d81d4894 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a7683a995ae4403b291afc021e4e0d3 in term 0.
I20260928 06:28:51.002072 30455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2932c651a3e047d484822aae9f61edb4" candidate_uuid: "3a7683a995ae4403b291afc021e4e0d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1813e3cd67549a1a161c3c54abc3c3b" is_pre_election: true
I20260928 06:28:51.002076 30526 leader_election.cc:302] T 2932c651a3e047d484822aae9f61edb4 P 3a7683a995ae4403b291afc021e4e0d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 377df16ac26d45a8b8114cc6d81d4894, 3a7683a995ae4403b291afc021e4e0d3; no voters:
I20260928 06:28:51.002158 30455 raft_consensus.cc:2454] T 2932c651a3e047d484822aae9f61edb4 P c1813e3cd67549a1a161c3c54abc3c3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a7683a995ae4403b291afc021e4e0d3 in term 0.
I20260928 06:28:51.002236 30870 raft_consensus.cc:2781] T 2932c651a3e047d484822aae9f61edb4 P 3a7683a995ae4403b291afc021e4e0d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:28:51.002282 30870 raft_consensus.cc:482] T 2932c651a3e047d484822aae9f61edb4 P 3a7683a995ae4403b291afc021e4e0d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:28:51.002307 30870 raft_consensus.cc:3037] T 2932c651a3e047d484822aae9f61edb4 P 3a7683a995ae4403b291afc021e4e0d3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:51.002979 30870 raft_consensus.cc:504] T 2932c651a3e047d484822aae9f61edb4 P 3a7683a995ae4403b291afc021e4e0d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1813e3cd67549a1a161c3c54abc3c3b" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 40749 } } peers { permanent_uuid: "3a7683a995ae4403b291afc021e4e0d3" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 43647 } } peers { permanent_uuid: "377df16ac26d45a8b8114cc6d81d4894" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 38727 } }
I20260928 06:28:51.003118 30870 leader_election.cc:288] T 2932c651a3e047d484822aae9f61edb4 P 3a7683a995ae4403b291afc021e4e0d3 [CANDIDATE]: Term 1 election: Requested vote from peers c1813e3cd67549a1a161c3c54abc3c3b (127.22.225.129:40749), 377df16ac26d45a8b8114cc6d81d4894 (127.22.225.131:38727)
I20260928 06:28:51.003273 30455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2932c651a3e047d484822aae9f61edb4" candidate_uuid: "3a7683a995ae4403b291afc021e4e0d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1813e3cd67549a1a161c3c54abc3c3b"
I20260928 06:28:51.003280 30728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2932c651a3e047d484822aae9f61edb4" candidate_uuid: "3a7683a995ae4403b291afc021e4e0d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "377df16ac26d45a8b8114cc6d81d4894"
I20260928 06:28:51.003340 30728 raft_consensus.cc:3037] T 2932c651a3e047d484822aae9f61edb4 P 377df16ac26d45a8b8114cc6d81d4894 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:51.003360 30455 raft_consensus.cc:3037] T 2932c651a3e047d484822aae9f61edb4 P c1813e3cd67549a1a161c3c54abc3c3b [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:51.003903 30455 raft_consensus.cc:2454] T 2932c651a3e047d484822aae9f61edb4 P c1813e3cd67549a1a161c3c54abc3c3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a7683a995ae4403b291afc021e4e0d3 in term 1.
I20260928 06:28:51.004109 30728 raft_consensus.cc:2454] T 2932c651a3e047d484822aae9f61edb4 P 377df16ac26d45a8b8114cc6d81d4894 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a7683a995ae4403b291afc021e4e0d3 in term 1.
I20260928 06:28:51.004112 30528 leader_election.cc:302] T 2932c651a3e047d484822aae9f61edb4 P 3a7683a995ae4403b291afc021e4e0d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a7683a995ae4403b291afc021e4e0d3, c1813e3cd67549a1a161c3c54abc3c3b; no voters:
I20260928 06:28:51.004218 30870 raft_consensus.cc:2781] T 2932c651a3e047d484822aae9f61edb4 P 3a7683a995ae4403b291afc021e4e0d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:28:51.004365 30870 raft_consensus.cc:686] T 2932c651a3e047d484822aae9f61edb4 P 3a7683a995ae4403b291afc021e4e0d3 [term 1 LEADER]: Becoming Leader. State: Replica: 3a7683a995ae4403b291afc021e4e0d3, State: Running, Role: LEADER
I20260928 06:28:51.004444 30870 consensus_queue.cc:237] T 2932c651a3e047d484822aae9f61edb4 P 3a7683a995ae4403b291afc021e4e0d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1813e3cd67549a1a161c3c54abc3c3b" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 40749 } } peers { permanent_uuid: "3a7683a995ae4403b291afc021e4e0d3" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 43647 } } peers { permanent_uuid: "377df16ac26d45a8b8114cc6d81d4894" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 38727 } }
I20260928 06:28:51.005014 30872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:51.004727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:41857 (local address 127.22.225.130:47415)
0928 06:28:51.004784 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:51.004984 (+ 200us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 06:28:51.005242 30137 catalog_manager.cc:5495] T 2932c651a3e047d484822aae9f61edb4 P 3a7683a995ae4403b291afc021e4e0d3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a7683a995ae4403b291afc021e4e0d3 (127.22.225.130). New cstate: current_term: 1 leader_uuid: "3a7683a995ae4403b291afc021e4e0d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1813e3cd67549a1a161c3c54abc3c3b" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 40749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a7683a995ae4403b291afc021e4e0d3" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 43647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "377df16ac26d45a8b8114cc6d81d4894" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 38727 } health_report { overall_health: UNKNOWN } } }
I20260928 06:28:51.031812 30864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:51.029403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60242 (local address 127.22.225.130:43647)
0928 06:28:51.029472 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:51.029474 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:51.029493 (+ 19us) server_negotiation.cc:407] Connection header received
0928 06:28:51.029544 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:51.029545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:51.029569 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:51.029648 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:28:51.029787 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:51.030276 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:51.031012 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:51.031212 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:51.031290 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:51.031449 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:28:51.031560 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:28:51.031638 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:51.031699 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 06:28:51.031721 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 06:28:51.032981 30455 raft_consensus.cc:1261] T 2932c651a3e047d484822aae9f61edb4 P c1813e3cd67549a1a161c3c54abc3c3b [term 1 FOLLOWER]: Refusing update from remote peer 3a7683a995ae4403b291afc021e4e0d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:28:51.033128 30728 raft_consensus.cc:1261] T 2932c651a3e047d484822aae9f61edb4 P 377df16ac26d45a8b8114cc6d81d4894 [term 1 FOLLOWER]: Refusing update from remote peer 3a7683a995ae4403b291afc021e4e0d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:28:51.033277 30879 consensus_queue.cc:1035] T 2932c651a3e047d484822aae9f61edb4 P 3a7683a995ae4403b291afc021e4e0d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1813e3cd67549a1a161c3c54abc3c3b" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 40749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:28:51.033386 30870 consensus_queue.cc:1035] T 2932c651a3e047d484822aae9f61edb4 P 3a7683a995ae4403b291afc021e4e0d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "377df16ac26d45a8b8114cc6d81d4894" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 38727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:28:51.034327 30877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:51.034119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:41857 (local address 127.22.225.131:49397)
0928 06:28:51.034254 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:51.034288 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260928 06:28:51.034471 30875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:51.034167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:41857 (local address 127.22.225.129:38977)
0928 06:28:51.034409 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:51.034438 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224}
I20260928 06:28:51.035454 30872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:51.035333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:41857 (local address 127.22.225.130:53361)
0928 06:28:51.035393 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:51.035429 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 06:28:51.036499 30872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:51.036362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:41857 (local address 127.22.225.130:46833)
0928 06:28:51.036447 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:51.036473 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260928 06:28:51.036661 30875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:51.036478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:41857 (local address 127.22.225.129:51739)
0928 06:28:51.036615 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:51.036640 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260928 06:28:51.037408 30136 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44640:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:28:51.037519 30136 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44640:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:28:51.038852 30877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:51.038595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:41857 (local address 127.22.225.131:38313)
0928 06:28:51.038775 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:51.038823 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165}
I20260928 06:28:51.038954 30136 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f15b6ec81d284f2baa79fa54469b144b: Sending DeleteTablet for 3 replicas of tablet 2932c651a3e047d484822aae9f61edb4
I20260928 06:28:51.039283 30708 tablet_service.cc:1514] Processing DeleteTablet for tablet 2932c651a3e047d484822aae9f61edb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:28:51 UTC) from {username='oryx', principal='oryx/127.22.225.189@KRBTEST.COM'} at 127.0.0.1:58996
I20260928 06:28:51.039332 30435 tablet_service.cc:1514] Processing DeleteTablet for tablet 2932c651a3e047d484822aae9f61edb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:28:51 UTC) from {username='oryx', principal='oryx/127.22.225.189@KRBTEST.COM'} at 127.0.0.1:35054
I20260928 06:28:51.039428 30572 tablet_service.cc:1514] Processing DeleteTablet for tablet 2932c651a3e047d484822aae9f61edb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:28:51 UTC) from {username='oryx', principal='oryx/127.22.225.189@KRBTEST.COM'} at 127.0.0.1:60236
I20260928 06:28:51.039532 30893 tablet_replica.cc:330] T 2932c651a3e047d484822aae9f61edb4 P c1813e3cd67549a1a161c3c54abc3c3b: stopping tablet replica
I20260928 06:28:51.039585 30893 raft_consensus.cc:2229] T 2932c651a3e047d484822aae9f61edb4 P c1813e3cd67549a1a161c3c54abc3c3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:28:51.039625 30893 raft_consensus.cc:2258] T 2932c651a3e047d484822aae9f61edb4 P c1813e3cd67549a1a161c3c54abc3c3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:28:51.039762 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 30284
I20260928 06:28:51.040885 30894 tablet_replica.cc:330] T 2932c651a3e047d484822aae9f61edb4 P 3a7683a995ae4403b291afc021e4e0d3: stopping tablet replica
I20260928 06:28:51.040940 30894 raft_consensus.cc:2229] T 2932c651a3e047d484822aae9f61edb4 P 3a7683a995ae4403b291afc021e4e0d3 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:28:51.041016 30894 raft_consensus.cc:2258] T 2932c651a3e047d484822aae9f61edb4 P 3a7683a995ae4403b291afc021e4e0d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:28:51.041253 30892 tablet_replica.cc:330] T 2932c651a3e047d484822aae9f61edb4 P 377df16ac26d45a8b8114cc6d81d4894: stopping tablet replica
I20260928 06:28:51.041312 30892 raft_consensus.cc:2229] T 2932c651a3e047d484822aae9f61edb4 P 377df16ac26d45a8b8114cc6d81d4894 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:28:51.041363 30892 raft_consensus.cc:2258] T 2932c651a3e047d484822aae9f61edb4 P 377df16ac26d45a8b8114cc6d81d4894 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:28:51.041559 30894 ts_tablet_manager.cc:1905] T 2932c651a3e047d484822aae9f61edb4 P 3a7683a995ae4403b291afc021e4e0d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:28:51.041676 30892 ts_tablet_manager.cc:1905] T 2932c651a3e047d484822aae9f61edb4 P 377df16ac26d45a8b8114cc6d81d4894: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:28:51.043182 30894 ts_tablet_manager.cc:1918] T 2932c651a3e047d484822aae9f61edb4 P 3a7683a995ae4403b291afc021e4e0d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:28:51.043247 30894 log.cc:1196] T 2932c651a3e047d484822aae9f61edb4 P 3a7683a995ae4403b291afc021e4e0d3: Deleting WAL directory at /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-1/wal/wals/2932c651a3e047d484822aae9f61edb4
I20260928 06:28:51.043498 30894 ts_tablet_manager.cc:1939] T 2932c651a3e047d484822aae9f61edb4 P 3a7683a995ae4403b291afc021e4e0d3: Deleting consensus metadata
I20260928 06:28:51.043865 30123 catalog_manager.cc:4841] TS 3a7683a995ae4403b291afc021e4e0d3 (127.22.225.130:43647): tablet 2932c651a3e047d484822aae9f61edb4 (table test-table [id=c0a616827b6c4a77827b1c41ede05886]) successfully deleted
I20260928 06:28:51.044319 30892 ts_tablet_manager.cc:1918] T 2932c651a3e047d484822aae9f61edb4 P 377df16ac26d45a8b8114cc6d81d4894: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:28:51.044381 30892 log.cc:1196] T 2932c651a3e047d484822aae9f61edb4 P 377df16ac26d45a8b8114cc6d81d4894: Deleting WAL directory at /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576899143097-23430-0/minicluster-data/ts-2/wal/wals/2932c651a3e047d484822aae9f61edb4
I20260928 06:28:51.044652 30892 ts_tablet_manager.cc:1939] T 2932c651a3e047d484822aae9f61edb4 P 377df16ac26d45a8b8114cc6d81d4894: Deleting consensus metadata
I20260928 06:28:51.044999 30122 catalog_manager.cc:4841] TS 377df16ac26d45a8b8114cc6d81d4894 (127.22.225.131:38727): tablet 2932c651a3e047d484822aae9f61edb4 (table test-table [id=c0a616827b6c4a77827b1c41ede05886]) successfully deleted
I20260928 06:28:51.047807 30861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:51.047671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.129:40749 (local address 127.0.0.1:35068)
0928 06:28:51.047738 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:51.047768 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.129:40749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 06:28:51.047963 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 30509
W20260928 06:28:51.047955 30124 catalog_manager.cc:4584] TS c1813e3cd67549a1a161c3c54abc3c3b (127.22.225.129:40749): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2932c651a3e047d484822aae9f61edb4: Network error: Client connection negotiation failed: client connection to 127.22.225.129:40749: connect: Connection refused (error 111)
I20260928 06:28:51.055605 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 30645
I20260928 06:28:51.061604 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 30002
I20260928 06:28:51.066803 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 30191
2026-09-28T06:28:51Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5326 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:28:51 dist-test-slave-d59m krb5kdc[30897](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:28:51 dist-test-slave-d59m krb5kdc[30897](info): set up 2 sockets
Sep 28 06:28:51 dist-test-slave-d59m krb5kdc[30897](info): commencing operation
krb5kdc: starting...
W20260928 06:28:52.113656 23430 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Sep 28 06:28:52 dist-test-slave-d59m krb5kdc[30897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:28:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:28:52Z Disabled control of system clock
WARNING: no policy specified for oryx/127.22.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/krb5kdc/oryx_127.22.225.190.keytab.
Entry for principal oryx/127.22.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/krb5kdc/oryx_127.22.225.190.keytab.
I20260928 06:28:52.239840 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:44991
--webserver_interface=127.22.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/krb5kdc/oryx_127.22.225.190.keytab
--principal=oryx/127.22.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:33541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.225.190:44991
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:28:52.322597 30910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:28:52.322811 30910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:28:52.322834 30910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:28:52.323963 30910 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:28:52.324007 30910 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 06:28:52.324021 30910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:28:52.324034 30910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:28:52.324046 30910 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:28:52.324059 30910 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:28:52.325709 30910 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:33541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.225.190:44991
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/krb5kdc/oryx_127.22.225.190.keytab
--principal=oryx/127.22.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:44991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30910
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:28:52.325973 30910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:28:52.326218 30910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:28:52.331097 30918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:52.331118 30916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:52.331197 30915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:52.331233 30910 server_base.cc:1034] running on GCE node
I20260928 06:28:52.331568 30910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:28:52.331887 30910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:28:52.333039 30910 hybrid_clock.cc:648] HybridClock initialized: now 1790576932333016 us; error 39 us; skew 500 ppm
Sep 28 06:28:52 dist-test-slave-d59m krb5kdc[30897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576932, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:28:52.335531 30910 init.cc:377] Logged in from keytab as oryx/127.22.225.190@KRBTEST.COM (short username oryx)
I20260928 06:28:52.336032 30910 webserver.cc:460] Webserver started at http://127.22.225.190:33987/ using document root <none> and password file <none>
I20260928 06:28:52.336285 30910 fs_manager.cc:359] Metadata directory not provided
I20260928 06:28:52.336329 30910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:28:52.336473 30910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:28:52.337558 30910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/master-0/data/instance:
uuid: "58908c52510e473e923860f490430124"
format_stamp: "Formatted at 2026-09-28 06:28:52 on dist-test-slave-d59m"
server_key: "b0969f851691fb0c5425e112ce1f6a4c"
server_key_iv: "8a7d903c2832588bf87cfbde52034de6"
server_key_version: "encryptionkey@0"
I20260928 06:28:52.337921 30910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/master-0/wal/instance:
uuid: "58908c52510e473e923860f490430124"
format_stamp: "Formatted at 2026-09-28 06:28:52 on dist-test-slave-d59m"
server_key: "b0969f851691fb0c5425e112ce1f6a4c"
server_key_iv: "8a7d903c2832588bf87cfbde52034de6"
server_key_version: "encryptionkey@0"
I20260928 06:28:52.339270 30910 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:28:52.340025 30925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:52.340178 30910 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:28:52.340267 30910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/master-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/master-0/wal
uuid: "58908c52510e473e923860f490430124"
format_stamp: "Formatted at 2026-09-28 06:28:52 on dist-test-slave-d59m"
server_key: "b0969f851691fb0c5425e112ce1f6a4c"
server_key_iv: "8a7d903c2832588bf87cfbde52034de6"
server_key_version: "encryptionkey@0"
I20260928 06:28:52.340337 30910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:28:52.356123 30910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:28:52.358253 30910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:28:52.358429 30910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:28:52.363632 30910 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.190:44991
I20260928 06:28:52.363709 30987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.190:44991 every 8 connection(s)
I20260928 06:28:52.364049 30910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
I20260928 06:28:52.364246 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 30910
I20260928 06:28:52.364346 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/master-0/wal/instance
I20260928 06:28:52.364570 23430 external_mini_cluster.cc:1428] Setting key 9abcb5af3cbbd1267e0fcb38e4354066
I20260928 06:28:52.364897 30988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:52.368615 30988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58908c52510e473e923860f490430124: Bootstrap starting.
I20260928 06:28:52.369391 30988 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58908c52510e473e923860f490430124: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:52.369752 30988 log.cc:824] T 00000000000000000000000000000000 P 58908c52510e473e923860f490430124: Log is configured to *not* fsync() on all Append() calls
I20260928 06:28:52.370633 30988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58908c52510e473e923860f490430124: No bootstrap required, opened a new log
Sep 28 06:28:52 dist-test-slave-d59m krb5kdc[30897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM
I20260928 06:28:52.373242 30988 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58908c52510e473e923860f490430124 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58908c52510e473e923860f490430124" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 44991 } }
I20260928 06:28:52.373423 30988 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58908c52510e473e923860f490430124 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:52.373488 30988 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58908c52510e473e923860f490430124 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58908c52510e473e923860f490430124, State: Initialized, Role: FOLLOWER
I20260928 06:28:52.373653 30988 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58908c52510e473e923860f490430124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58908c52510e473e923860f490430124" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 44991 } }
I20260928 06:28:52.373744 30988 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58908c52510e473e923860f490430124 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:28:52.373781 30988 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58908c52510e473e923860f490430124 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:28:52.373831 30988 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58908c52510e473e923860f490430124 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:52.374539 30988 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58908c52510e473e923860f490430124 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58908c52510e473e923860f490430124" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 44991 } }
I20260928 06:28:52.374665 30988 leader_election.cc:302] T 00000000000000000000000000000000 P 58908c52510e473e923860f490430124 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58908c52510e473e923860f490430124; no voters:
I20260928 06:28:52.374825 30988 leader_election.cc:288] T 00000000000000000000000000000000 P 58908c52510e473e923860f490430124 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:28:52.374907 30993 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58908c52510e473e923860f490430124 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:28:52.375061 30988 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58908c52510e473e923860f490430124 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:28:52.375066 30993 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58908c52510e473e923860f490430124 [term 1 LEADER]: Becoming Leader. State: Replica: 58908c52510e473e923860f490430124, State: Running, Role: LEADER
I20260928 06:28:52.375298 30993 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58908c52510e473e923860f490430124 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58908c52510e473e923860f490430124" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 44991 } }
I20260928 06:28:52.376310 30991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:52.365185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53002 (local address 127.22.225.190:44991)
0928 06:28:52.365382 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:52.365391 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:52.365421 (+ 30us) server_negotiation.cc:407] Connection header received
0928 06:28:52.366146 (+ 725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:52.366153 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:52.366201 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:52.366356 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:52.366951 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:52.367872 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:52.368675 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:52.369123 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:52.369244 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:52.372318 (+ 3074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:52.372322 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:52.372323 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:52.372335 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:52.374458 (+ 2123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:52.374860 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:52.374863 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:52.374864 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:52.374980 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:52.375313 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:52.375314 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:52.375315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:52.375643 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:52.375824 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:52.375907 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 06:28:52.376051 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":122,"threads_started":1}
I20260928 06:28:52.376515 30995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58908c52510e473e923860f490430124 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58908c52510e473e923860f490430124. Latest consensus state: current_term: 1 leader_uuid: "58908c52510e473e923860f490430124" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58908c52510e473e923860f490430124" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 44991 } } }
I20260928 06:28:52.376627 30995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58908c52510e473e923860f490430124 [sys.catalog]: This master's current role is: LEADER
I20260928 06:28:52.376617 30994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58908c52510e473e923860f490430124 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58908c52510e473e923860f490430124" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58908c52510e473e923860f490430124" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 44991 } } }
I20260928 06:28:52.377063 30994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58908c52510e473e923860f490430124 [sys.catalog]: This master's current role is: LEADER
W20260928 06:28:52.377076 31008 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 58908c52510e473e923860f490430124: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:28:52.377183 31008 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:28:52.377401 31010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:28:52.377570 31010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:28:52.379485 31010 catalog_manager.cc:1347] Generated new cluster ID: 3b150ed3af464e578a6c7037fd3df603
I20260928 06:28:52.379540 31010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:28:52.387717 31010 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:28:52.388411 31010 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:28:52.394994 31010 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58908c52510e473e923860f490430124: Generated new TSK 0
I20260928 06:28:52.395174 31010 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.22.225.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.225.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.225.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/krb5kdc/oryx_127.22.225.129.keytab.
Entry for principal oryx/127.22.225.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/krb5kdc/oryx_127.22.225.129.keytab.
I20260928 06:28:52.431459 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.129:0
--local_ip_for_outbound_sockets=127.22.225.129
--webserver_interface=127.22.225.129
--webserver_port=0
--tserver_master_addrs=127.22.225.190:44991
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/krb5kdc/oryx_127.22.225.129.keytab
--principal=oryx/127.22.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:33541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:28:52.523291 31014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:28:52.523542 31014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:28:52.523587 31014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:28:52.524907 31014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:28:52.524969 31014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:28:52.525007 31014 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 06:28:52.525077 31014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.129
I20260928 06:28:52.526834 31014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:33541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/krb5kdc/oryx_127.22.225.129.keytab
--principal=oryx/127.22.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.225.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.225.190:44991
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.129
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:28:52.527119 31014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:28:52.527451 31014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:28:52.533313 31020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:52.533329 31019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:52.533540 31022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:52.533769 31014 server_base.cc:1034] running on GCE node
I20260928 06:28:52.533981 31014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:28:52.534269 31014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:28:52.535499 31014 hybrid_clock.cc:648] HybridClock initialized: now 1790576932535476 us; error 34 us; skew 500 ppm
Sep 28 06:28:52 dist-test-slave-d59m krb5kdc[30897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576932, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:28:52.538810 31014 init.cc:377] Logged in from keytab as oryx/127.22.225.129@KRBTEST.COM (short username oryx)
I20260928 06:28:52.539398 31014 webserver.cc:460] Webserver started at http://127.22.225.129:46053/ using document root <none> and password file <none>
I20260928 06:28:52.539680 31014 fs_manager.cc:359] Metadata directory not provided
I20260928 06:28:52.539744 31014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:28:52.539901 31014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:28:52.541440 31014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/ts-0/data/instance:
uuid: "555fe426f3e5471ba94d6e86c80bda01"
format_stamp: "Formatted at 2026-09-28 06:28:52 on dist-test-slave-d59m"
server_key: "8e6bb17212915bd1bf220b75bf15e50b"
server_key_iv: "5dd4e4fcf29b5ac5f3cd93b4483eaa5b"
server_key_version: "encryptionkey@0"
I20260928 06:28:52.541888 31014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/ts-0/wal/instance:
uuid: "555fe426f3e5471ba94d6e86c80bda01"
format_stamp: "Formatted at 2026-09-28 06:28:52 on dist-test-slave-d59m"
server_key: "8e6bb17212915bd1bf220b75bf15e50b"
server_key_iv: "5dd4e4fcf29b5ac5f3cd93b4483eaa5b"
server_key_version: "encryptionkey@0"
I20260928 06:28:52.543787 31014 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:28:52.544723 31029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:52.544893 31014 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:28:52.544996 31014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/ts-0/wal
uuid: "555fe426f3e5471ba94d6e86c80bda01"
format_stamp: "Formatted at 2026-09-28 06:28:52 on dist-test-slave-d59m"
server_key: "8e6bb17212915bd1bf220b75bf15e50b"
server_key_iv: "5dd4e4fcf29b5ac5f3cd93b4483eaa5b"
server_key_version: "encryptionkey@0"
I20260928 06:28:52.545065 31014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:28:52.567638 31014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:28:52.569919 31014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:28:52.570098 31014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:28:52.570616 31014 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:28:52.570679 31014 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:52.570724 31014 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:28:52.570752 31014 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:52.578024 31014 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.129:39737
I20260928 06:28:52.578114 31144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.129:39737 every 8 connection(s)
Sep 28 06:28:52 dist-test-slave-d59m krb5kdc[30897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576932, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.225.129@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM
I20260928 06:28:52.578426 31014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
I20260928 06:28:52.581831 30991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:52.571162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:35739 (local address 127.22.225.190:44991)
0928 06:28:52.571203 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:52.571205 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:52.571505 (+ 300us) server_negotiation.cc:407] Connection header received
0928 06:28:52.572185 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:52.572187 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:52.572200 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:52.572274 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:52.573156 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:52.574008 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:52.575583 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:52.575926 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:52.575991 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:52.578905 (+ 2914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:52.578906 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:52.578907 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:52.578914 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:52.580435 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:52.580870 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:52.580872 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:52.580872 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:52.580941 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:52.581197 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:52.581198 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:52.581198 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:52.581337 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:52.581407 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:52.581647 (+ 240us) server_negotiation.cc:299] Negotiation successful
0928 06:28:52.581736 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:28:52.581851 31038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:52.571163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:44991 (local address 127.22.225.129:35739)
0928 06:28:52.571318 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:52.571366 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:52.572052 (+ 686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:52.572344 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:52.572349 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:52.572745 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:52.573049 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:52.573057 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:52.574109 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:52.574110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:52.575483 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:52.575486 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:52.576056 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:52.576057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:52.576207 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:52.577033 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:52.577043 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:52.578788 (+ 1745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:52.580517 (+ 1729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:52.580523 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:52.580525 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:52.580800 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:52.581009 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:52.581010 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:52.581010 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:52.581124 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:52.581401 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:52.581405 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:52.581514 (+ 109us) client_negotiation.cc:769] Sending connection context
0928 06:28:52.581587 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 06:28:52.581699 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":81,"threads_started":1}
I20260928 06:28:52.582482 31145 heartbeater.cc:347] Connected to a master server at 127.22.225.190:44991
I20260928 06:28:52.582587 31145 heartbeater.cc:464] Registering TS with master...
I20260928 06:28:52.582770 31145 heartbeater.cc:511] Master 127.22.225.190:44991 requested a full tablet report, sending...
I20260928 06:28:52.583649 30941 ts_manager.cc:194] Registered new tserver with Master: 555fe426f3e5471ba94d6e86c80bda01 (127.22.225.129:39737)
I20260928 06:28:52.584574 30941 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.225.129@KRBTEST.COM'} at 127.22.225.129:35739
I20260928 06:28:52.588125 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 31014
I20260928 06:28:52.588265 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/ts-0/wal/instance
I20260928 06:28:52.588466 23430 external_mini_cluster.cc:1428] Setting key a4419b5838bb71fb9508215f953fcf21
I20260928 06:28:52.590227 30991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:52.583573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:56157 (local address 127.22.225.190:44991)
0928 06:28:52.584815 (+ 1242us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:52.584817 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:52.584825 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:28:52.584843 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:52.584843 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:52.584852 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:52.584915 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:52.585271 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:52.585734 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:52.586460 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:52.586655 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:52.586739 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:52.587524 (+ 785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:52.587525 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:52.587526 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:52.587533 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:52.589158 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:52.589568 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:52.589569 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:52.589569 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:52.589619 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:52.589815 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:52.589816 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:52.589816 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:52.589904 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:52.589953 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:52.590118 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 06:28:52.590171 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1225}
I20260928 06:28:52.590382 31151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:52.583525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:44991 (local address 127.22.225.129:56157)
0928 06:28:52.583653 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:52.583663 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:52.583726 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:52.584933 (+ 1207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:52.584934 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:52.585134 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:52.585190 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:52.585192 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:52.585841 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:52.585842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:52.586365 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:52.586366 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:52.586741 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:52.586743 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:52.586813 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:52.587186 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:52.587190 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:52.587435 (+ 245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:52.589249 (+ 1814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:52.589250 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:52.589250 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:52.589483 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:52.589682 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:52.589683 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:52.589683 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:52.589755 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:52.589968 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:52.589970 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:52.590020 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 06:28:52.590101 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 06:28:52.590219 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":77,"threads_started":1}
WARNING: no policy specified for oryx/127.22.225.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.225.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.225.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/krb5kdc/oryx_127.22.225.130.keytab.
Entry for principal oryx/127.22.225.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/krb5kdc/oryx_127.22.225.130.keytab.
I20260928 06:28:52.615142 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.130:0
--local_ip_for_outbound_sockets=127.22.225.130
--webserver_interface=127.22.225.130
--webserver_port=0
--tserver_master_addrs=127.22.225.190:44991
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/krb5kdc/oryx_127.22.225.130.keytab
--principal=oryx/127.22.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:33541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:28:52.712963 31154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:28:52.713253 31154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:28:52.713287 31154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:28:52.714924 31154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:28:52.714983 31154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:28:52.715024 31154 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 06:28:52.715060 31154 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.130
I20260928 06:28:52.717422 31154 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:33541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/krb5kdc/oryx_127.22.225.130.keytab
--principal=oryx/127.22.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.225.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.225.190:44991
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31154
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.130
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:28:52.717670 31154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:28:52.717921 31154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:28:52.723362 31159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:52.723465 31162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:52.723366 31160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:52.724215 31154 server_base.cc:1034] running on GCE node
I20260928 06:28:52.724432 31154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:28:52.724702 31154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:28:52.725883 31154 hybrid_clock.cc:648] HybridClock initialized: now 1790576932725867 us; error 31 us; skew 500 ppm
Sep 28 06:28:52 dist-test-slave-d59m krb5kdc[30897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576932, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:28:52.729140 31154 init.cc:377] Logged in from keytab as oryx/127.22.225.130@KRBTEST.COM (short username oryx)
I20260928 06:28:52.729637 31154 webserver.cc:460] Webserver started at http://127.22.225.130:33669/ using document root <none> and password file <none>
I20260928 06:28:52.729943 31154 fs_manager.cc:359] Metadata directory not provided
I20260928 06:28:52.730029 31154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:28:52.730132 31154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:28:52.731478 31154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/ts-1/data/instance:
uuid: "1b186b7e369049d9a87566312419b111"
format_stamp: "Formatted at 2026-09-28 06:28:52 on dist-test-slave-d59m"
server_key: "558cf4312f0b11b84ac72a648dce1833"
server_key_iv: "d5964cb897d62338ee194ee7455e038f"
server_key_version: "encryptionkey@0"
I20260928 06:28:52.731796 31154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/ts-1/wal/instance:
uuid: "1b186b7e369049d9a87566312419b111"
format_stamp: "Formatted at 2026-09-28 06:28:52 on dist-test-slave-d59m"
server_key: "558cf4312f0b11b84ac72a648dce1833"
server_key_iv: "d5964cb897d62338ee194ee7455e038f"
server_key_version: "encryptionkey@0"
I20260928 06:28:52.733886 31154 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260928 06:28:52.735239 31169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:52.735406 31154 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:28:52.735502 31154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/ts-1/wal
uuid: "1b186b7e369049d9a87566312419b111"
format_stamp: "Formatted at 2026-09-28 06:28:52 on dist-test-slave-d59m"
server_key: "558cf4312f0b11b84ac72a648dce1833"
server_key_iv: "d5964cb897d62338ee194ee7455e038f"
server_key_version: "encryptionkey@0"
I20260928 06:28:52.735592 31154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:28:52.754315 31154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:28:52.756808 31154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:28:52.756963 31154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:28:52.757499 31154 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:28:52.757565 31154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:52.757606 31154 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:28:52.757632 31154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:52.764979 31154 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.130:41181
Sep 28 06:28:52 dist-test-slave-d59m krb5kdc[30897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576932, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.225.130@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM
I20260928 06:28:52.765053 31284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.130:41181 every 8 connection(s)
I20260928 06:28:52.765441 31154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
I20260928 06:28:52.768486 30991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:52.757985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:42007 (local address 127.22.225.190:44991)
0928 06:28:52.758028 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:52.758030 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:52.758352 (+ 322us) server_negotiation.cc:407] Connection header received
0928 06:28:52.758889 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:52.758891 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:52.758905 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:52.758997 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:52.759893 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:52.760773 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:52.762557 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:52.762901 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:52.762981 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:52.765618 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:52.765620 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:52.765620 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:52.765628 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:52.767077 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:52.767485 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:52.767486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:52.767487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:52.767546 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:52.767798 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:52.767799 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:52.767799 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:52.767956 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:52.768030 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:52.768310 (+ 280us) server_negotiation.cc:299] Negotiation successful
0928 06:28:52.768393 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 06:28:52.768712 31178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:52.758002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:44991 (local address 127.22.225.130:42007)
0928 06:28:52.758150 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:52.758195 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:52.758784 (+ 589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:52.759104 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:52.759109 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:52.759465 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:52.759769 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:52.759777 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:52.760942 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:52.760943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:52.762395 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:52.762399 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:52.762987 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:52.762987 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:52.763093 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:52.763693 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:52.763702 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:52.765507 (+ 1805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:52.767173 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:52.767180 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:52.767182 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:52.767406 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:52.767611 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:52.767611 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:52.767612 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:52.767719 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:52.768024 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:52.768031 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:52.768171 (+ 140us) client_negotiation.cc:769] Sending connection context
0928 06:28:52.768288 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 06:28:52.768448 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":81,"threads_started":1}
I20260928 06:28:52.769456 31285 heartbeater.cc:347] Connected to a master server at 127.22.225.190:44991
I20260928 06:28:52.769558 31285 heartbeater.cc:464] Registering TS with master...
I20260928 06:28:52.769797 31285 heartbeater.cc:511] Master 127.22.225.190:44991 requested a full tablet report, sending...
I20260928 06:28:52.770262 30941 ts_manager.cc:194] Registered new tserver with Master: 1b186b7e369049d9a87566312419b111 (127.22.225.130:41181)
I20260928 06:28:52.770782 30941 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.225.130@KRBTEST.COM'} at 127.22.225.130:42007
I20260928 06:28:52.771154 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 31154
I20260928 06:28:52.771283 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/ts-1/wal/instance
I20260928 06:28:52.771520 23430 external_mini_cluster.cc:1428] Setting key 7fa6de1b05213b9260ed004ea7e43219
I20260928 06:28:52.777611 30991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:52.770618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:49675 (local address 127.22.225.190:44991)
0928 06:28:52.770658 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:52.770659 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:52.770747 (+ 88us) server_negotiation.cc:407] Connection header received
0928 06:28:52.770824 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:52.770825 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:52.770834 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:52.770896 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:52.771333 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:52.772075 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:52.772909 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:52.773138 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:52.773223 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:52.774512 (+ 1289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:52.774513 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:52.774513 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:52.774519 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:52.776208 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:52.776650 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:52.776651 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:52.776652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:52.776732 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:52.776956 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:52.776957 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:52.776957 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:52.777162 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:52.777235 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:52.777424 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 06:28:52.777507 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:28:52.777644 31291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:52.770528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:44991 (local address 127.22.225.130:49675)
0928 06:28:52.770676 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:52.770687 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:52.770772 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:52.770916 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:52.770917 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:52.771152 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:52.771239 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:52.771241 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:52.772217 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:52.772218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:52.772830 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:52.772832 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:52.773226 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:52.773227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:52.773296 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:52.774112 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:52.774117 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:52.774387 (+ 270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:52.776309 (+ 1922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:52.776310 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:52.776311 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:52.776525 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:52.776807 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:52.776808 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:52.776809 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:52.776885 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:52.777245 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:52.777246 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:52.777302 (+ 56us) client_negotiation.cc:769] Sending connection context
0928 06:28:52.777385 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 06:28:52.777498 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":85,"threads_started":1}
WARNING: no policy specified for oryx/127.22.225.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.225.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.225.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/krb5kdc/oryx_127.22.225.131.keytab.
Entry for principal oryx/127.22.225.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/krb5kdc/oryx_127.22.225.131.keytab.
I20260928 06:28:52.799217 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.131:0
--local_ip_for_outbound_sockets=127.22.225.131
--webserver_interface=127.22.225.131
--webserver_port=0
--tserver_master_addrs=127.22.225.190:44991
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/krb5kdc/oryx_127.22.225.131.keytab
--principal=oryx/127.22.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:33541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:28:52.885049 31294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:28:52.885265 31294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:28:52.885291 31294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:28:52.886482 31294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:28:52.886521 31294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:28:52.886543 31294 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 06:28:52.886560 31294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.131
I20260928 06:28:52.888074 31294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:33541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/krb5kdc/oryx_127.22.225.131.keytab
--principal=oryx/127.22.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.225.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.225.190:44991
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.131
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:28:52.888319 31294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:28:52.888567 31294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:28:52.893541 31302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:52.893575 31299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:52.893577 31300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:52.894132 31294 server_base.cc:1034] running on GCE node
I20260928 06:28:52.894382 31294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:28:52.894622 31294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:28:52.895777 31294 hybrid_clock.cc:648] HybridClock initialized: now 1790576932895766 us; error 36 us; skew 500 ppm
Sep 28 06:28:52 dist-test-slave-d59m krb5kdc[30897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576932, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:28:52.898386 31294 init.cc:377] Logged in from keytab as oryx/127.22.225.131@KRBTEST.COM (short username oryx)
I20260928 06:28:52.898837 31294 webserver.cc:460] Webserver started at http://127.22.225.131:35241/ using document root <none> and password file <none>
I20260928 06:28:52.899077 31294 fs_manager.cc:359] Metadata directory not provided
I20260928 06:28:52.899148 31294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:28:52.899297 31294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:28:52.900349 31294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/ts-2/data/instance:
uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd"
format_stamp: "Formatted at 2026-09-28 06:28:52 on dist-test-slave-d59m"
server_key: "76b27d452315333adeeb19c69adad308"
server_key_iv: "c0c3275474a358c6463a309ffe237c68"
server_key_version: "encryptionkey@0"
I20260928 06:28:52.900740 31294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/ts-2/wal/instance:
uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd"
format_stamp: "Formatted at 2026-09-28 06:28:52 on dist-test-slave-d59m"
server_key: "76b27d452315333adeeb19c69adad308"
server_key_iv: "c0c3275474a358c6463a309ffe237c68"
server_key_version: "encryptionkey@0"
I20260928 06:28:52.902118 31294 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:28:52.902915 31309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:52.903100 31294 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:28:52.903199 31294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/ts-2/wal
uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd"
format_stamp: "Formatted at 2026-09-28 06:28:52 on dist-test-slave-d59m"
server_key: "76b27d452315333adeeb19c69adad308"
server_key_iv: "c0c3275474a358c6463a309ffe237c68"
server_key_version: "encryptionkey@0"
I20260928 06:28:52.903277 31294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:28:52.914008 31294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:28:52.916950 31294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:28:52.917124 31294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:28:52.917745 31294 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:28:52.917834 31294 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:52.917901 31294 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:28:52.917950 31294 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 06:28:52 dist-test-slave-d59m krb5kdc[30897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576932, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.225.131@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM
I20260928 06:28:52.925423 31294 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.131:44361
I20260928 06:28:52.925833 31294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
I20260928 06:28:52.926388 31424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.131:44361 every 8 connection(s)
I20260928 06:28:52.930033 30991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:52.918111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:53903 (local address 127.22.225.190:44991)
0928 06:28:52.918164 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:52.918166 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:52.918440 (+ 274us) server_negotiation.cc:407] Connection header received
0928 06:28:52.919201 (+ 761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:52.919203 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:52.919216 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:52.919294 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:52.920154 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:52.921041 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:52.922623 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:52.922972 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:52.923054 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:52.925916 (+ 2862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:52.925917 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:52.925918 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:52.925926 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:52.928550 (+ 2624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:52.928968 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:52.928970 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:52.928970 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:52.929077 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:52.929371 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:52.929372 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:52.929373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:52.929492 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:52.929545 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:52.929871 (+ 326us) server_negotiation.cc:299] Negotiation successful
0928 06:28:52.929971 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 06:28:52.930245 31318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:52.918113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:44991 (local address 127.22.225.131:53903)
0928 06:28:52.918270 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:52.918325 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:52.919077 (+ 752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:52.919376 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:52.919381 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:52.919761 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:52.920051 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:52.920058 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:52.921169 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:52.921170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:52.922522 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:52.922525 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:52.923124 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:52.923125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:52.923266 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:52.923885 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:52.923896 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:52.925809 (+ 1913us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:52.928670 (+ 2861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:52.928675 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:52.928677 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:52.928883 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:52.929167 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:52.929168 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:52.929169 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:52.929302 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:52.929560 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:52.929566 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:52.929723 (+ 157us) client_negotiation.cc:769] Sending connection context
0928 06:28:52.929843 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 06:28:52.930006 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":78,"threads_started":1}
I20260928 06:28:52.930810 31425 heartbeater.cc:347] Connected to a master server at 127.22.225.190:44991
I20260928 06:28:52.930931 31425 heartbeater.cc:464] Registering TS with master...
I20260928 06:28:52.931164 31425 heartbeater.cc:511] Master 127.22.225.190:44991 requested a full tablet report, sending...
I20260928 06:28:52.931628 30942 ts_manager.cc:194] Registered new tserver with Master: 6569a7bc6c3a4ab9819f125bcf4fbffd (127.22.225.131:44361)
I20260928 06:28:52.932144 30942 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.225.131@KRBTEST.COM'} at 127.22.225.131:53903
I20260928 06:28:52.934401 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 31294
I20260928 06:28:52.934510 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/ts-2/wal/instance
I20260928 06:28:52.934717 23430 external_mini_cluster.cc:1428] Setting key 5c98576f093f1910f4c133ecb0f0f922
I20260928 06:28:52.935866 23430 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:28:52.938068 30991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:52.932302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:36769 (local address 127.22.225.190:44991)
0928 06:28:52.932336 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:52.932338 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:52.932345 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:28:52.932362 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:52.932363 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:52.932371 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:52.932450 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:52.932845 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:52.933303 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:52.934014 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:52.934232 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:52.934305 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:52.935311 (+ 1006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:52.935313 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:52.935313 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:52.935319 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:52.936804 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:52.937208 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:52.937209 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:52.937209 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:52.937271 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:52.937485 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:52.937486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:52.937487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:52.937626 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:52.937695 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:52.937888 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 06:28:52.937965 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 06:28:52.938356 31431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:52.931828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:44991 (local address 127.22.225.131:36769)
0928 06:28:52.931950 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:52.931960 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:52.932009 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:52.932456 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:52.932457 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:52.932707 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:52.932760 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:52.932762 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:52.933398 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:52.933399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:52.933906 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:52.933908 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:52.934304 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:52.934305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:52.934371 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:52.934818 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:52.934823 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:52.935225 (+ 402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:52.936889 (+ 1664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:52.936890 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:52.936890 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:52.937126 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:52.937343 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:52.937344 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:52.937345 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:52.937413 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:52.937727 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:52.937728 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:52.937772 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 06:28:52.938148 (+ 376us) client_negotiation.cc:240] Negotiation successful
0928 06:28:52.938251 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":69,"threads_started":1}
Sep 28 06:28:52 dist-test-slave-d59m krb5kdc[30897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576932, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 06:28:52 dist-test-slave-d59m krb5kdc[30897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:28:52 dist-test-slave-d59m krb5kdc[30897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
W20260928 06:28:52.958865 30991 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:28:52.952499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53016 (local address 127.22.225.190:44991)
0928 06:28:52.952543 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:52.952545 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:52.952619 (+ 74us) server_negotiation.cc:407] Connection header received
0928 06:28:52.952715 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:52.952716 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:52.952725 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:52.952775 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:52.953468 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:52.953936 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:52.955004 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:52.955179 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:52.955228 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:52.958773 (+ 3545us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53016: BlockingRecv error: recv got EOF from 127.0.0.1:53016 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 28 06:28:52 dist-test-slave-d59m krb5kdc[30897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576932, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM
I20260928 06:28:52.966176 30991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:52.959741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53024 (local address 127.22.225.190:44991)
0928 06:28:52.959807 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:52.959808 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:52.959900 (+ 92us) server_negotiation.cc:407] Connection header received
0928 06:28:52.959981 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:52.959982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:52.959994 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:52.960057 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:52.960608 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:52.961087 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:52.961773 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:52.961968 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:52.962055 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:52.964073 (+ 2018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:52.964075 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:52.964075 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:52.964081 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:52.965304 (+ 1223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:52.965594 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:52.965595 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:52.965595 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:52.965643 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:52.965811 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:52.965812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:52.965812 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:52.965902 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:52.965974 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:52.966062 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 06:28:52.966115 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 06:28:52.967866 30942 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53024:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:28:52.969839 30942 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:28:52.977339 31452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:52.972528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.130:41181 (local address 127.0.0.1:57146)
0928 06:28:52.973034 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:52.973045 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:52.973106 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:52.973337 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:52.973338 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:52.973340 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:28:52.973395 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:52.973397 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:52.974742 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:52.974743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:52.976002 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:52.976005 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:52.977059 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:52.977060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:52.977135 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:52.977148 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 06:28:52.977215 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 06:28:52.977261 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":451,"threads_started":1}
I20260928 06:28:52.977665 31447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:52.972345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.129:39737 (local address 127.0.0.1:55466)
0928 06:28:52.972793 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:52.972803 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:52.972842 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:52.973499 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:52.973500 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:52.973501 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:28:52.973542 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:52.973543 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:52.975418 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:52.975419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:52.976491 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:52.976493 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:52.977464 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:52.977465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:52.977527 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:52.977532 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:28:52.977563 (+ 31us) client_negotiation.cc:240] Negotiation successful
0928 06:28:52.977595 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":79,"threads_started":1}
I20260928 06:28:52.977682 31451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:52.972826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57146 (local address 127.22.225.130:41181)
0928 06:28:52.972983 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:52.972987 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:52.973130 (+ 143us) server_negotiation.cc:407] Connection header received
0928 06:28:52.973173 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:52.973175 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:52.973191 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:52.973262 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:28:52.973501 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:52.974647 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:52.976096 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:52.976891 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:52.976973 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:52.977026 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:52.977552 (+ 526us) server_negotiation.cc:299] Negotiation successful
0928 06:28:52.977593 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":80,"threads_started":1}
I20260928 06:28:52.977910 31448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:52.972338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.131:44361 (local address 127.0.0.1:35664)
0928 06:28:52.972506 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:52.972547 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:52.972688 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:52.973204 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:52.973208 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:52.973210 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:28:52.973273 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:52.973275 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:52.974576 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:52.974577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:52.976057 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:52.976060 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:52.977676 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:52.977678 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:52.977747 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:52.977751 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:28:52.977782 (+ 31us) client_negotiation.cc:240] Negotiation successful
0928 06:28:52.977822 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":39,"thread_start_us":57,"threads_started":1}
I20260928 06:28:52.978010 31450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:52.972743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55466 (local address 127.22.225.129:39737)
0928 06:28:52.973326 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:52.973330 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:52.973339 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:28:52.973376 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:52.973379 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:52.973392 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:52.973611 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:28:52.973666 (+ 55us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:52.975328 (+ 1662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:52.976636 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:52.977365 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:52.977844 (+ 479us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:52.977885 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:52.977906 (+ 21us) server_negotiation.cc:299] Negotiation successful
0928 06:28:52.977933 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":93,"threads_started":1}
I20260928 06:28:52.978605 31219 tablet_service.cc:1467] Processing CreateTablet for tablet 665ceee924d449cc916bd510c215c0e0 (DEFAULT_TABLE table=test-table [id=9d1bf26420fa4d1c9fae8eb8f6831220]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:28:52.978879 31219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 665ceee924d449cc916bd510c215c0e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:52.978735 31079 tablet_service.cc:1467] Processing CreateTablet for tablet 665ceee924d449cc916bd510c215c0e0 (DEFAULT_TABLE table=test-table [id=9d1bf26420fa4d1c9fae8eb8f6831220]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:28:52.979096 31079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 665ceee924d449cc916bd510c215c0e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:52.981846 31453 tablet_bootstrap.cc:492] T 665ceee924d449cc916bd510c215c0e0 P 1b186b7e369049d9a87566312419b111: Bootstrap starting.
I20260928 06:28:52.982503 31453 tablet_bootstrap.cc:654] T 665ceee924d449cc916bd510c215c0e0 P 1b186b7e369049d9a87566312419b111: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:52.982864 31453 log.cc:824] T 665ceee924d449cc916bd510c215c0e0 P 1b186b7e369049d9a87566312419b111: Log is configured to *not* fsync() on all Append() calls
I20260928 06:28:52.983568 31453 tablet_bootstrap.cc:492] T 665ceee924d449cc916bd510c215c0e0 P 1b186b7e369049d9a87566312419b111: No bootstrap required, opened a new log
I20260928 06:28:52.983649 31453 ts_tablet_manager.cc:1397] T 665ceee924d449cc916bd510c215c0e0 P 1b186b7e369049d9a87566312419b111: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:28:52.984048 31455 tablet_bootstrap.cc:492] T 665ceee924d449cc916bd510c215c0e0 P 555fe426f3e5471ba94d6e86c80bda01: Bootstrap starting.
I20260928 06:28:52.984458 31449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:52.972430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35664 (local address 127.22.225.131:44361)
0928 06:28:52.972957 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:52.972963 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:52.972983 (+ 20us) server_negotiation.cc:407] Connection header received
0928 06:28:52.973023 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:52.973025 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:52.973041 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:52.973197 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:28:52.973358 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:52.974478 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:52.976476 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:52.977325 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:52.984211 (+ 6886us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:52.984285 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:52.984313 (+ 28us) server_negotiation.cc:299] Negotiation successful
0928 06:28:52.984362 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":106,"threads_started":1}
I20260928 06:28:52.985213 31455 tablet_bootstrap.cc:654] T 665ceee924d449cc916bd510c215c0e0 P 555fe426f3e5471ba94d6e86c80bda01: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:52.985162 31359 tablet_service.cc:1467] Processing CreateTablet for tablet 665ceee924d449cc916bd510c215c0e0 (DEFAULT_TABLE table=test-table [id=9d1bf26420fa4d1c9fae8eb8f6831220]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:28:52.985543 31359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 665ceee924d449cc916bd510c215c0e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:52.985679 31455 log.cc:824] T 665ceee924d449cc916bd510c215c0e0 P 555fe426f3e5471ba94d6e86c80bda01: Log is configured to *not* fsync() on all Append() calls
I20260928 06:28:52.986517 31455 tablet_bootstrap.cc:492] T 665ceee924d449cc916bd510c215c0e0 P 555fe426f3e5471ba94d6e86c80bda01: No bootstrap required, opened a new log
I20260928 06:28:52.986629 31455 ts_tablet_manager.cc:1397] T 665ceee924d449cc916bd510c215c0e0 P 555fe426f3e5471ba94d6e86c80bda01: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 06:28:52.989195 31457 tablet_bootstrap.cc:492] T 665ceee924d449cc916bd510c215c0e0 P 6569a7bc6c3a4ab9819f125bcf4fbffd: Bootstrap starting.
I20260928 06:28:52.989151 31455 raft_consensus.cc:348] T 665ceee924d449cc916bd510c215c0e0 P 555fe426f3e5471ba94d6e86c80bda01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 44361 } } peers { permanent_uuid: "555fe426f3e5471ba94d6e86c80bda01" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 39737 } } peers { permanent_uuid: "1b186b7e369049d9a87566312419b111" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41181 } }
I20260928 06:28:52.989387 31455 raft_consensus.cc:374] T 665ceee924d449cc916bd510c215c0e0 P 555fe426f3e5471ba94d6e86c80bda01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:52.989434 31455 raft_consensus.cc:729] T 665ceee924d449cc916bd510c215c0e0 P 555fe426f3e5471ba94d6e86c80bda01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 555fe426f3e5471ba94d6e86c80bda01, State: Initialized, Role: FOLLOWER
I20260928 06:28:52.989552 31455 consensus_queue.cc:260] T 665ceee924d449cc916bd510c215c0e0 P 555fe426f3e5471ba94d6e86c80bda01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 44361 } } peers { permanent_uuid: "555fe426f3e5471ba94d6e86c80bda01" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 39737 } } peers { permanent_uuid: "1b186b7e369049d9a87566312419b111" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41181 } }
I20260928 06:28:52.989820 31457 tablet_bootstrap.cc:654] T 665ceee924d449cc916bd510c215c0e0 P 6569a7bc6c3a4ab9819f125bcf4fbffd: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:52.989842 31455 ts_tablet_manager.cc:1428] T 665ceee924d449cc916bd510c215c0e0 P 555fe426f3e5471ba94d6e86c80bda01: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:28:52.989778 31453 raft_consensus.cc:348] T 665ceee924d449cc916bd510c215c0e0 P 1b186b7e369049d9a87566312419b111 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 44361 } } peers { permanent_uuid: "555fe426f3e5471ba94d6e86c80bda01" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 39737 } } peers { permanent_uuid: "1b186b7e369049d9a87566312419b111" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41181 } }
I20260928 06:28:52.989945 31453 raft_consensus.cc:374] T 665ceee924d449cc916bd510c215c0e0 P 1b186b7e369049d9a87566312419b111 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:52.989853 31145 heartbeater.cc:503] Master 127.22.225.190:44991 was elected leader, sending a full tablet report...
I20260928 06:28:52.989996 31453 raft_consensus.cc:729] T 665ceee924d449cc916bd510c215c0e0 P 1b186b7e369049d9a87566312419b111 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b186b7e369049d9a87566312419b111, State: Initialized, Role: FOLLOWER
I20260928 06:28:52.990113 31453 consensus_queue.cc:260] T 665ceee924d449cc916bd510c215c0e0 P 1b186b7e369049d9a87566312419b111 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 44361 } } peers { permanent_uuid: "555fe426f3e5471ba94d6e86c80bda01" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 39737 } } peers { permanent_uuid: "1b186b7e369049d9a87566312419b111" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41181 } }
I20260928 06:28:52.990160 31457 log.cc:824] T 665ceee924d449cc916bd510c215c0e0 P 6569a7bc6c3a4ab9819f125bcf4fbffd: Log is configured to *not* fsync() on all Append() calls
I20260928 06:28:52.990386 31453 ts_tablet_manager.cc:1428] T 665ceee924d449cc916bd510c215c0e0 P 1b186b7e369049d9a87566312419b111: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20260928 06:28:52.990506 31285 heartbeater.cc:503] Master 127.22.225.190:44991 was elected leader, sending a full tablet report...
I20260928 06:28:52.991052 31457 tablet_bootstrap.cc:492] T 665ceee924d449cc916bd510c215c0e0 P 6569a7bc6c3a4ab9819f125bcf4fbffd: No bootstrap required, opened a new log
I20260928 06:28:52.991130 31457 ts_tablet_manager.cc:1397] T 665ceee924d449cc916bd510c215c0e0 P 6569a7bc6c3a4ab9819f125bcf4fbffd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:28:52.992632 31457 raft_consensus.cc:348] T 665ceee924d449cc916bd510c215c0e0 P 6569a7bc6c3a4ab9819f125bcf4fbffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 44361 } } peers { permanent_uuid: "555fe426f3e5471ba94d6e86c80bda01" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 39737 } } peers { permanent_uuid: "1b186b7e369049d9a87566312419b111" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41181 } }
I20260928 06:28:52.992792 31457 raft_consensus.cc:374] T 665ceee924d449cc916bd510c215c0e0 P 6569a7bc6c3a4ab9819f125bcf4fbffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:52.992835 31457 raft_consensus.cc:729] T 665ceee924d449cc916bd510c215c0e0 P 6569a7bc6c3a4ab9819f125bcf4fbffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6569a7bc6c3a4ab9819f125bcf4fbffd, State: Initialized, Role: FOLLOWER
I20260928 06:28:52.992952 31457 consensus_queue.cc:260] T 665ceee924d449cc916bd510c215c0e0 P 6569a7bc6c3a4ab9819f125bcf4fbffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 44361 } } peers { permanent_uuid: "555fe426f3e5471ba94d6e86c80bda01" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 39737 } } peers { permanent_uuid: "1b186b7e369049d9a87566312419b111" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41181 } }
I20260928 06:28:52.993208 31457 ts_tablet_manager.cc:1428] T 665ceee924d449cc916bd510c215c0e0 P 6569a7bc6c3a4ab9819f125bcf4fbffd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:28:52.993238 31425 heartbeater.cc:503] Master 127.22.225.190:44991 was elected leader, sending a full tablet report...
W20260928 06:28:53.016196 31286 tablet.cc:2369] T 665ceee924d449cc916bd510c215c0e0 P 1b186b7e369049d9a87566312419b111: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:28:53.079181 31146 tablet.cc:2369] T 665ceee924d449cc916bd510c215c0e0 P 555fe426f3e5471ba94d6e86c80bda01: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:28:53.164045 31459 raft_consensus.cc:482] T 665ceee924d449cc916bd510c215c0e0 P 1b186b7e369049d9a87566312419b111 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:28:53.164167 31459 raft_consensus.cc:504] T 665ceee924d449cc916bd510c215c0e0 P 1b186b7e369049d9a87566312419b111 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 44361 } } peers { permanent_uuid: "555fe426f3e5471ba94d6e86c80bda01" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 39737 } } peers { permanent_uuid: "1b186b7e369049d9a87566312419b111" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41181 } }
I20260928 06:28:53.164569 31459 leader_election.cc:288] T 665ceee924d449cc916bd510c215c0e0 P 1b186b7e369049d9a87566312419b111 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6569a7bc6c3a4ab9819f125bcf4fbffd (127.22.225.131:44361), 555fe426f3e5471ba94d6e86c80bda01 (127.22.225.129:39737)
I20260928 06:28:53.167644 31450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:53.164871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:47325 (local address 127.22.225.129:39737)
0928 06:28:53.164915 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:53.164917 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:53.165185 (+ 268us) server_negotiation.cc:407] Connection header received
0928 06:28:53.165256 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:53.165257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:53.165266 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:53.165336 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:28:53.165509 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:53.165980 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:53.166925 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:53.167310 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:53.167381 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:53.167389 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:53.167536 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 06:28:53.167575 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:28:53.167837 31462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:53.164782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.129:39737 (local address 127.22.225.130:47325)
0928 06:28:53.165104 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:53.165122 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:53.165192 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:53.165340 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:53.165341 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:53.165342 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:28:53.165421 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:53.165423 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:53.166068 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:53.166069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:53.166830 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:53.166832 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:53.167377 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:53.167378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:53.167463 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:53.167467 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:28:53.167516 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 06:28:53.167554 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":147,"thread_start_us":105,"threads_started":1}
I20260928 06:28:53.168274 31178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:53.164778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.131:44361 (local address 127.22.225.130:37033)
0928 06:28:53.164839 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:53.164852 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:53.164892 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:53.165068 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:53.165069 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:53.165071 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:28:53.165146 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:53.165147 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:53.166095 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:53.166096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:53.167225 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:53.167227 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:53.168029 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:53.168030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:53.168109 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:53.168112 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:28:53.168169 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 06:28:53.168206 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 06:28:53.168288 31449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:53.164870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:37033 (local address 127.22.225.131:44361)
0928 06:28:53.164915 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:53.164917 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:53.164926 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:28:53.164957 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:53.164958 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:53.164979 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:53.165046 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:28:53.165235 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:53.165989 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:53.167322 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:53.167900 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:53.167965 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:53.167973 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:53.168187 (+ 214us) server_negotiation.cc:299] Negotiation successful
0928 06:28:53.168229 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:28:53.168285 31099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "665ceee924d449cc916bd510c215c0e0" candidate_uuid: "1b186b7e369049d9a87566312419b111" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "555fe426f3e5471ba94d6e86c80bda01" is_pre_election: true
I20260928 06:28:53.168457 31099 raft_consensus.cc:2454] T 665ceee924d449cc916bd510c215c0e0 P 555fe426f3e5471ba94d6e86c80bda01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b186b7e369049d9a87566312419b111 in term 0.
I20260928 06:28:53.168556 31379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "665ceee924d449cc916bd510c215c0e0" candidate_uuid: "1b186b7e369049d9a87566312419b111" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" is_pre_election: true
I20260928 06:28:53.168731 31379 raft_consensus.cc:2454] T 665ceee924d449cc916bd510c215c0e0 P 6569a7bc6c3a4ab9819f125bcf4fbffd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b186b7e369049d9a87566312419b111 in term 0.
I20260928 06:28:53.168736 31171 leader_election.cc:302] T 665ceee924d449cc916bd510c215c0e0 P 1b186b7e369049d9a87566312419b111 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b186b7e369049d9a87566312419b111, 555fe426f3e5471ba94d6e86c80bda01; no voters:
I20260928 06:28:53.168860 31459 raft_consensus.cc:2781] T 665ceee924d449cc916bd510c215c0e0 P 1b186b7e369049d9a87566312419b111 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:28:53.168921 31459 raft_consensus.cc:482] T 665ceee924d449cc916bd510c215c0e0 P 1b186b7e369049d9a87566312419b111 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:28:53.168987 31459 raft_consensus.cc:3037] T 665ceee924d449cc916bd510c215c0e0 P 1b186b7e369049d9a87566312419b111 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:53.169734 31459 raft_consensus.cc:504] T 665ceee924d449cc916bd510c215c0e0 P 1b186b7e369049d9a87566312419b111 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 44361 } } peers { permanent_uuid: "555fe426f3e5471ba94d6e86c80bda01" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 39737 } } peers { permanent_uuid: "1b186b7e369049d9a87566312419b111" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41181 } }
I20260928 06:28:53.169983 31459 leader_election.cc:288] T 665ceee924d449cc916bd510c215c0e0 P 1b186b7e369049d9a87566312419b111 [CANDIDATE]: Term 1 election: Requested vote from peers 6569a7bc6c3a4ab9819f125bcf4fbffd (127.22.225.131:44361), 555fe426f3e5471ba94d6e86c80bda01 (127.22.225.129:39737)
I20260928 06:28:53.170068 31379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "665ceee924d449cc916bd510c215c0e0" candidate_uuid: "1b186b7e369049d9a87566312419b111" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd"
I20260928 06:28:53.170120 31099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "665ceee924d449cc916bd510c215c0e0" candidate_uuid: "1b186b7e369049d9a87566312419b111" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "555fe426f3e5471ba94d6e86c80bda01"
I20260928 06:28:53.170145 31379 raft_consensus.cc:3037] T 665ceee924d449cc916bd510c215c0e0 P 6569a7bc6c3a4ab9819f125bcf4fbffd [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:53.170193 31099 raft_consensus.cc:3037] T 665ceee924d449cc916bd510c215c0e0 P 555fe426f3e5471ba94d6e86c80bda01 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:53.171010 31379 raft_consensus.cc:2454] T 665ceee924d449cc916bd510c215c0e0 P 6569a7bc6c3a4ab9819f125bcf4fbffd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b186b7e369049d9a87566312419b111 in term 1.
I20260928 06:28:53.171010 31099 raft_consensus.cc:2454] T 665ceee924d449cc916bd510c215c0e0 P 555fe426f3e5471ba94d6e86c80bda01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b186b7e369049d9a87566312419b111 in term 1.
I20260928 06:28:53.171162 31171 leader_election.cc:302] T 665ceee924d449cc916bd510c215c0e0 P 1b186b7e369049d9a87566312419b111 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b186b7e369049d9a87566312419b111, 555fe426f3e5471ba94d6e86c80bda01; no voters:
I20260928 06:28:53.171254 31459 raft_consensus.cc:2781] T 665ceee924d449cc916bd510c215c0e0 P 1b186b7e369049d9a87566312419b111 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:28:53.171444 31459 raft_consensus.cc:686] T 665ceee924d449cc916bd510c215c0e0 P 1b186b7e369049d9a87566312419b111 [term 1 LEADER]: Becoming Leader. State: Replica: 1b186b7e369049d9a87566312419b111, State: Running, Role: LEADER
I20260928 06:28:53.171553 31459 consensus_queue.cc:237] T 665ceee924d449cc916bd510c215c0e0 P 1b186b7e369049d9a87566312419b111 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 44361 } } peers { permanent_uuid: "555fe426f3e5471ba94d6e86c80bda01" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 39737 } } peers { permanent_uuid: "1b186b7e369049d9a87566312419b111" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41181 } }
I20260928 06:28:53.172698 30942 catalog_manager.cc:5495] T 665ceee924d449cc916bd510c215c0e0 P 1b186b7e369049d9a87566312419b111 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b186b7e369049d9a87566312419b111 (127.22.225.130). New cstate: current_term: 1 leader_uuid: "1b186b7e369049d9a87566312419b111" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 44361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "555fe426f3e5471ba94d6e86c80bda01" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 39737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b186b7e369049d9a87566312419b111" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41181 } health_report { overall_health: HEALTHY } } }
W20260928 06:28:53.176754 31426 tablet.cc:2369] T 665ceee924d449cc916bd510c215c0e0 P 6569a7bc6c3a4ab9819f125bcf4fbffd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:28:53.210215 31451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:53.207148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57148 (local address 127.22.225.130:41181)
0928 06:28:53.207220 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:53.207222 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:53.207232 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:28:53.207279 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:53.207280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:53.207303 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:53.207395 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:28:53.207572 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:53.208382 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:53.209187 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:53.209534 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:53.209616 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:53.209790 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:28:53.209943 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:28:53.210046 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:53.210065 (+ 19us) server_negotiation.cc:299] Negotiation successful
0928 06:28:53.210093 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 06:28:53.212039 31379 raft_consensus.cc:1261] T 665ceee924d449cc916bd510c215c0e0 P 6569a7bc6c3a4ab9819f125bcf4fbffd [term 1 FOLLOWER]: Refusing update from remote peer 1b186b7e369049d9a87566312419b111: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:28:53.212100 31099 raft_consensus.cc:1261] T 665ceee924d449cc916bd510c215c0e0 P 555fe426f3e5471ba94d6e86c80bda01 [term 1 FOLLOWER]: Refusing update from remote peer 1b186b7e369049d9a87566312419b111: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:28:53.212436 31459 consensus_queue.cc:1035] T 665ceee924d449cc916bd510c215c0e0 P 1b186b7e369049d9a87566312419b111 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 44361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:28:53.212638 31463 consensus_queue.cc:1035] T 665ceee924d449cc916bd510c215c0e0 P 1b186b7e369049d9a87566312419b111 [LEADER]: Connected to new peer: Peer: permanent_uuid: "555fe426f3e5471ba94d6e86c80bda01" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 39737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:28:53.217531 30941 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:28:53.217631 30941 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:28:53.218437 30941 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 58908c52510e473e923860f490430124: Sending DeleteTablet for 3 replicas of tablet 665ceee924d449cc916bd510c215c0e0
I20260928 06:28:53.218770 31359 tablet_service.cc:1514] Processing DeleteTablet for tablet 665ceee924d449cc916bd510c215c0e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:28:53 UTC) from {username='oryx', principal='oryx/127.22.225.190@KRBTEST.COM'} at 127.0.0.1:35664
I20260928 06:28:53.218796 31079 tablet_service.cc:1514] Processing DeleteTablet for tablet 665ceee924d449cc916bd510c215c0e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:28:53 UTC) from {username='oryx', principal='oryx/127.22.225.190@KRBTEST.COM'} at 127.0.0.1:55466
I20260928 06:28:53.219053 31477 tablet_replica.cc:330] T 665ceee924d449cc916bd510c215c0e0 P 555fe426f3e5471ba94d6e86c80bda01: stopping tablet replica
I20260928 06:28:53.219039 31219 tablet_service.cc:1514] Processing DeleteTablet for tablet 665ceee924d449cc916bd510c215c0e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:28:53 UTC) from {username='oryx', principal='oryx/127.22.225.190@KRBTEST.COM'} at 127.0.0.1:57146
I20260928 06:28:53.219017 30941 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53024:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 06:28:53.219157 31477 raft_consensus.cc:2229] T 665ceee924d449cc916bd510c215c0e0 P 555fe426f3e5471ba94d6e86c80bda01 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260928 06:28:53.219170 30941 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:28:53.219261 31477 raft_consensus.cc:2258] T 665ceee924d449cc916bd510c215c0e0 P 555fe426f3e5471ba94d6e86c80bda01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:28:53.220316 31219 tablet_service.cc:1467] Processing CreateTablet for tablet aae5a108c9e543e2ac36a98cf4c989da (DEFAULT_TABLE table=test-table [id=ee810257885c4240bfd23fd8e844828e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:28:53.220322 31079 tablet_service.cc:1467] Processing CreateTablet for tablet aae5a108c9e543e2ac36a98cf4c989da (DEFAULT_TABLE table=test-table [id=ee810257885c4240bfd23fd8e844828e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:28:53.220348 31359 tablet_service.cc:1467] Processing CreateTablet for tablet aae5a108c9e543e2ac36a98cf4c989da (DEFAULT_TABLE table=test-table [id=ee810257885c4240bfd23fd8e844828e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:28:53.220443 31219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aae5a108c9e543e2ac36a98cf4c989da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:53.220445 31079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aae5a108c9e543e2ac36a98cf4c989da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:53.220443 31359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aae5a108c9e543e2ac36a98cf4c989da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:53.221827 31453 tablet_bootstrap.cc:492] T aae5a108c9e543e2ac36a98cf4c989da P 1b186b7e369049d9a87566312419b111: Bootstrap starting.
I20260928 06:28:53.221979 31457 tablet_bootstrap.cc:492] T aae5a108c9e543e2ac36a98cf4c989da P 6569a7bc6c3a4ab9819f125bcf4fbffd: Bootstrap starting.
I20260928 06:28:53.222149 31455 tablet_bootstrap.cc:492] T aae5a108c9e543e2ac36a98cf4c989da P 555fe426f3e5471ba94d6e86c80bda01: Bootstrap starting.
I20260928 06:28:53.222254 31453 tablet_bootstrap.cc:654] T aae5a108c9e543e2ac36a98cf4c989da P 1b186b7e369049d9a87566312419b111: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:53.222391 31457 tablet_bootstrap.cc:654] T aae5a108c9e543e2ac36a98cf4c989da P 6569a7bc6c3a4ab9819f125bcf4fbffd: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:53.222535 31455 tablet_bootstrap.cc:654] T aae5a108c9e543e2ac36a98cf4c989da P 555fe426f3e5471ba94d6e86c80bda01: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:53.223133 31457 tablet_bootstrap.cc:492] T aae5a108c9e543e2ac36a98cf4c989da P 6569a7bc6c3a4ab9819f125bcf4fbffd: No bootstrap required, opened a new log
I20260928 06:28:53.223133 31455 tablet_bootstrap.cc:492] T aae5a108c9e543e2ac36a98cf4c989da P 555fe426f3e5471ba94d6e86c80bda01: No bootstrap required, opened a new log
I20260928 06:28:53.223186 31455 ts_tablet_manager.cc:1397] T aae5a108c9e543e2ac36a98cf4c989da P 555fe426f3e5471ba94d6e86c80bda01: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:28:53.223186 31457 ts_tablet_manager.cc:1397] T aae5a108c9e543e2ac36a98cf4c989da P 6569a7bc6c3a4ab9819f125bcf4fbffd: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:28:53.223345 31455 raft_consensus.cc:348] T aae5a108c9e543e2ac36a98cf4c989da P 555fe426f3e5471ba94d6e86c80bda01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555fe426f3e5471ba94d6e86c80bda01" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 39737 } } peers { permanent_uuid: "1b186b7e369049d9a87566312419b111" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41181 } } peers { permanent_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 44361 } }
I20260928 06:28:53.223345 31457 raft_consensus.cc:348] T aae5a108c9e543e2ac36a98cf4c989da P 6569a7bc6c3a4ab9819f125bcf4fbffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555fe426f3e5471ba94d6e86c80bda01" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 39737 } } peers { permanent_uuid: "1b186b7e369049d9a87566312419b111" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41181 } } peers { permanent_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 44361 } }
I20260928 06:28:53.223417 31455 raft_consensus.cc:374] T aae5a108c9e543e2ac36a98cf4c989da P 555fe426f3e5471ba94d6e86c80bda01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:53.223417 31457 raft_consensus.cc:374] T aae5a108c9e543e2ac36a98cf4c989da P 6569a7bc6c3a4ab9819f125bcf4fbffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:53.223440 31457 raft_consensus.cc:729] T aae5a108c9e543e2ac36a98cf4c989da P 6569a7bc6c3a4ab9819f125bcf4fbffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6569a7bc6c3a4ab9819f125bcf4fbffd, State: Initialized, Role: FOLLOWER
I20260928 06:28:53.223439 31455 raft_consensus.cc:729] T aae5a108c9e543e2ac36a98cf4c989da P 555fe426f3e5471ba94d6e86c80bda01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 555fe426f3e5471ba94d6e86c80bda01, State: Initialized, Role: FOLLOWER
I20260928 06:28:53.223484 31455 consensus_queue.cc:260] T aae5a108c9e543e2ac36a98cf4c989da P 555fe426f3e5471ba94d6e86c80bda01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555fe426f3e5471ba94d6e86c80bda01" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 39737 } } peers { permanent_uuid: "1b186b7e369049d9a87566312419b111" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41181 } } peers { permanent_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 44361 } }
I20260928 06:28:53.223493 31457 consensus_queue.cc:260] T aae5a108c9e543e2ac36a98cf4c989da P 6569a7bc6c3a4ab9819f125bcf4fbffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555fe426f3e5471ba94d6e86c80bda01" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 39737 } } peers { permanent_uuid: "1b186b7e369049d9a87566312419b111" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41181 } } peers { permanent_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 44361 } }
I20260928 06:28:53.223582 31457 ts_tablet_manager.cc:1428] T aae5a108c9e543e2ac36a98cf4c989da P 6569a7bc6c3a4ab9819f125bcf4fbffd: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 06:28:53.223596 31455 ts_tablet_manager.cc:1428] T aae5a108c9e543e2ac36a98cf4c989da P 555fe426f3e5471ba94d6e86c80bda01: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 06:28:53.224001 31477 ts_tablet_manager.cc:1905] T 665ceee924d449cc916bd510c215c0e0 P 555fe426f3e5471ba94d6e86c80bda01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:28:53.225133 31478 tablet_replica.cc:330] T 665ceee924d449cc916bd510c215c0e0 P 1b186b7e369049d9a87566312419b111: stopping tablet replica
I20260928 06:28:53.225238 31478 raft_consensus.cc:2229] T 665ceee924d449cc916bd510c215c0e0 P 1b186b7e369049d9a87566312419b111 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:28:53.225324 31478 raft_consensus.cc:2258] T 665ceee924d449cc916bd510c215c0e0 P 1b186b7e369049d9a87566312419b111 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:28:53.225673 31477 ts_tablet_manager.cc:1918] T 665ceee924d449cc916bd510c215c0e0 P 555fe426f3e5471ba94d6e86c80bda01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:28:53.225744 31477 log.cc:1196] T 665ceee924d449cc916bd510c215c0e0 P 555fe426f3e5471ba94d6e86c80bda01: Deleting WAL directory at /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/ts-0/wal/wals/665ceee924d449cc916bd510c215c0e0
I20260928 06:28:53.225780 31478 ts_tablet_manager.cc:1905] T 665ceee924d449cc916bd510c215c0e0 P 1b186b7e369049d9a87566312419b111: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:28:53.226402 31477 ts_tablet_manager.cc:1939] T 665ceee924d449cc916bd510c215c0e0 P 555fe426f3e5471ba94d6e86c80bda01: Deleting consensus metadata
I20260928 06:28:53.226831 30927 catalog_manager.cc:4841] TS 555fe426f3e5471ba94d6e86c80bda01 (127.22.225.129:39737): tablet 665ceee924d449cc916bd510c215c0e0 (table test-table [id=9d1bf26420fa4d1c9fae8eb8f6831220]) successfully deleted
I20260928 06:28:53.227272 31478 ts_tablet_manager.cc:1918] T 665ceee924d449cc916bd510c215c0e0 P 1b186b7e369049d9a87566312419b111: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:28:53.227339 31478 log.cc:1196] T 665ceee924d449cc916bd510c215c0e0 P 1b186b7e369049d9a87566312419b111: Deleting WAL directory at /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/ts-1/wal/wals/665ceee924d449cc916bd510c215c0e0
I20260928 06:28:53.227612 31478 ts_tablet_manager.cc:1939] T 665ceee924d449cc916bd510c215c0e0 P 1b186b7e369049d9a87566312419b111: Deleting consensus metadata
I20260928 06:28:53.227938 31453 tablet_bootstrap.cc:492] T aae5a108c9e543e2ac36a98cf4c989da P 1b186b7e369049d9a87566312419b111: No bootstrap required, opened a new log
I20260928 06:28:53.227993 31453 ts_tablet_manager.cc:1397] T aae5a108c9e543e2ac36a98cf4c989da P 1b186b7e369049d9a87566312419b111: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260928 06:28:53.228014 30927 catalog_manager.cc:4841] TS 1b186b7e369049d9a87566312419b111 (127.22.225.130:41181): tablet 665ceee924d449cc916bd510c215c0e0 (table test-table [id=9d1bf26420fa4d1c9fae8eb8f6831220]) successfully deleted
I20260928 06:28:53.228148 31453 raft_consensus.cc:348] T aae5a108c9e543e2ac36a98cf4c989da P 1b186b7e369049d9a87566312419b111 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555fe426f3e5471ba94d6e86c80bda01" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 39737 } } peers { permanent_uuid: "1b186b7e369049d9a87566312419b111" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41181 } } peers { permanent_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 44361 } }
I20260928 06:28:53.228230 31453 raft_consensus.cc:374] T aae5a108c9e543e2ac36a98cf4c989da P 1b186b7e369049d9a87566312419b111 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:53.228257 31453 raft_consensus.cc:729] T aae5a108c9e543e2ac36a98cf4c989da P 1b186b7e369049d9a87566312419b111 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b186b7e369049d9a87566312419b111, State: Initialized, Role: FOLLOWER
I20260928 06:28:53.228315 31453 consensus_queue.cc:260] T aae5a108c9e543e2ac36a98cf4c989da P 1b186b7e369049d9a87566312419b111 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555fe426f3e5471ba94d6e86c80bda01" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 39737 } } peers { permanent_uuid: "1b186b7e369049d9a87566312419b111" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41181 } } peers { permanent_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 44361 } }
I20260928 06:28:53.228425 31453 ts_tablet_manager.cc:1428] T aae5a108c9e543e2ac36a98cf4c989da P 1b186b7e369049d9a87566312419b111: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:53.228664 31476 tablet_replica.cc:330] T 665ceee924d449cc916bd510c215c0e0 P 6569a7bc6c3a4ab9819f125bcf4fbffd: stopping tablet replica
I20260928 06:28:53.228783 31476 raft_consensus.cc:2229] T 665ceee924d449cc916bd510c215c0e0 P 6569a7bc6c3a4ab9819f125bcf4fbffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:28:53.228905 31476 raft_consensus.cc:2258] T 665ceee924d449cc916bd510c215c0e0 P 6569a7bc6c3a4ab9819f125bcf4fbffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:28:53.229787 31476 ts_tablet_manager.cc:1905] T 665ceee924d449cc916bd510c215c0e0 P 6569a7bc6c3a4ab9819f125bcf4fbffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:28:53.231765 31476 ts_tablet_manager.cc:1918] T 665ceee924d449cc916bd510c215c0e0 P 6569a7bc6c3a4ab9819f125bcf4fbffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:28:53.231834 31476 log.cc:1196] T 665ceee924d449cc916bd510c215c0e0 P 6569a7bc6c3a4ab9819f125bcf4fbffd: Deleting WAL directory at /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/ts-2/wal/wals/665ceee924d449cc916bd510c215c0e0
I20260928 06:28:53.232056 31476 ts_tablet_manager.cc:1939] T 665ceee924d449cc916bd510c215c0e0 P 6569a7bc6c3a4ab9819f125bcf4fbffd: Deleting consensus metadata
I20260928 06:28:53.232558 30929 catalog_manager.cc:4841] TS 6569a7bc6c3a4ab9819f125bcf4fbffd (127.22.225.131:44361): tablet 665ceee924d449cc916bd510c215c0e0 (table test-table [id=9d1bf26420fa4d1c9fae8eb8f6831220]) successfully deleted
I20260928 06:28:53.233527 31458 raft_consensus.cc:482] T aae5a108c9e543e2ac36a98cf4c989da P 555fe426f3e5471ba94d6e86c80bda01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:28:53.233593 31458 raft_consensus.cc:504] T aae5a108c9e543e2ac36a98cf4c989da P 555fe426f3e5471ba94d6e86c80bda01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555fe426f3e5471ba94d6e86c80bda01" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 39737 } } peers { permanent_uuid: "1b186b7e369049d9a87566312419b111" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41181 } } peers { permanent_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 44361 } }
I20260928 06:28:53.233867 31458 leader_election.cc:288] T aae5a108c9e543e2ac36a98cf4c989da P 555fe426f3e5471ba94d6e86c80bda01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b186b7e369049d9a87566312419b111 (127.22.225.130:41181), 6569a7bc6c3a4ab9819f125bcf4fbffd (127.22.225.131:44361)
I20260928 06:28:53.237984 31451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:53.234556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:49643 (local address 127.22.225.130:41181)
0928 06:28:53.234673 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:53.234675 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:53.234682 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:28:53.234703 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:53.234704 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:53.234712 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:53.234755 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:28:53.234914 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:53.235645 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:53.237076 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:53.237621 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:53.237668 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:53.237677 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:53.237886 (+ 209us) server_negotiation.cc:299] Negotiation successful
0928 06:28:53.237923 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260928 06:28:53.238057 31449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:53.234231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:58719 (local address 127.22.225.131:44361)
0928 06:28:53.234267 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:53.234269 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:53.234372 (+ 103us) server_negotiation.cc:407] Connection header received
0928 06:28:53.234432 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:53.234433 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:53.234442 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:53.234500 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:28:53.234631 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:53.235378 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:53.237119 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:53.237682 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:53.237727 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:53.237734 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:53.237959 (+ 225us) server_negotiation.cc:299] Negotiation successful
0928 06:28:53.237995 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:28:53.238221 31482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:53.234045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.130:41181 (local address 127.22.225.129:49643)
0928 06:28:53.234204 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:53.234216 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:53.234265 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:53.234790 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:53.234791 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:53.234793 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:28:53.234850 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:53.234851 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:53.235743 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:53.235744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:53.236976 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:53.236978 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:53.237734 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:53.237735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:53.237811 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:53.237814 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:28:53.238094 (+ 280us) client_negotiation.cc:240] Negotiation successful
0928 06:28:53.238136 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":61,"threads_started":1}
I20260928 06:28:53.238466 31239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aae5a108c9e543e2ac36a98cf4c989da" candidate_uuid: "555fe426f3e5471ba94d6e86c80bda01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b186b7e369049d9a87566312419b111" is_pre_election: true
I20260928 06:28:53.238607 31239 raft_consensus.cc:2454] T aae5a108c9e543e2ac36a98cf4c989da P 1b186b7e369049d9a87566312419b111 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 555fe426f3e5471ba94d6e86c80bda01 in term 0.
I20260928 06:28:53.238686 31483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:53.234074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.131:44361 (local address 127.22.225.129:58719)
0928 06:28:53.234313 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:53.234323 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:53.234382 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:53.234502 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:53.234503 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:53.234504 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:28:53.234580 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:53.234582 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:53.235478 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:53.235479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:53.237016 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:53.237019 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:53.237811 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:53.237812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:53.237881 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:53.237885 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:28:53.238533 (+ 648us) client_negotiation.cc:240] Negotiation successful
0928 06:28:53.238606 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":76,"threads_started":1}
I20260928 06:28:53.238794 31031 leader_election.cc:302] T aae5a108c9e543e2ac36a98cf4c989da P 555fe426f3e5471ba94d6e86c80bda01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b186b7e369049d9a87566312419b111, 555fe426f3e5471ba94d6e86c80bda01; no voters:
I20260928 06:28:53.238896 31458 raft_consensus.cc:2781] T aae5a108c9e543e2ac36a98cf4c989da P 555fe426f3e5471ba94d6e86c80bda01 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:28:53.238921 31458 raft_consensus.cc:482] T aae5a108c9e543e2ac36a98cf4c989da P 555fe426f3e5471ba94d6e86c80bda01 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:28:53.238888 31379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aae5a108c9e543e2ac36a98cf4c989da" candidate_uuid: "555fe426f3e5471ba94d6e86c80bda01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" is_pre_election: true
I20260928 06:28:53.238938 31458 raft_consensus.cc:3037] T aae5a108c9e543e2ac36a98cf4c989da P 555fe426f3e5471ba94d6e86c80bda01 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:53.238957 31379 raft_consensus.cc:2454] T aae5a108c9e543e2ac36a98cf4c989da P 6569a7bc6c3a4ab9819f125bcf4fbffd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 555fe426f3e5471ba94d6e86c80bda01 in term 0.
I20260928 06:28:53.239629 31458 raft_consensus.cc:504] T aae5a108c9e543e2ac36a98cf4c989da P 555fe426f3e5471ba94d6e86c80bda01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555fe426f3e5471ba94d6e86c80bda01" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 39737 } } peers { permanent_uuid: "1b186b7e369049d9a87566312419b111" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41181 } } peers { permanent_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 44361 } }
I20260928 06:28:53.239753 31458 leader_election.cc:288] T aae5a108c9e543e2ac36a98cf4c989da P 555fe426f3e5471ba94d6e86c80bda01 [CANDIDATE]: Term 1 election: Requested vote from peers 1b186b7e369049d9a87566312419b111 (127.22.225.130:41181), 6569a7bc6c3a4ab9819f125bcf4fbffd (127.22.225.131:44361)
I20260928 06:28:53.239909 31379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aae5a108c9e543e2ac36a98cf4c989da" candidate_uuid: "555fe426f3e5471ba94d6e86c80bda01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd"
I20260928 06:28:53.239964 31379 raft_consensus.cc:3037] T aae5a108c9e543e2ac36a98cf4c989da P 6569a7bc6c3a4ab9819f125bcf4fbffd [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:53.240134 31239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aae5a108c9e543e2ac36a98cf4c989da" candidate_uuid: "555fe426f3e5471ba94d6e86c80bda01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b186b7e369049d9a87566312419b111"
I20260928 06:28:53.240195 31239 raft_consensus.cc:3037] T aae5a108c9e543e2ac36a98cf4c989da P 1b186b7e369049d9a87566312419b111 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:53.240693 31379 raft_consensus.cc:2454] T aae5a108c9e543e2ac36a98cf4c989da P 6569a7bc6c3a4ab9819f125bcf4fbffd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 555fe426f3e5471ba94d6e86c80bda01 in term 1.
I20260928 06:28:53.240885 31033 leader_election.cc:302] T aae5a108c9e543e2ac36a98cf4c989da P 555fe426f3e5471ba94d6e86c80bda01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 555fe426f3e5471ba94d6e86c80bda01, 6569a7bc6c3a4ab9819f125bcf4fbffd; no voters:
I20260928 06:28:53.241082 31239 raft_consensus.cc:2454] T aae5a108c9e543e2ac36a98cf4c989da P 1b186b7e369049d9a87566312419b111 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 555fe426f3e5471ba94d6e86c80bda01 in term 1.
I20260928 06:28:53.241353 31458 raft_consensus.cc:2781] T aae5a108c9e543e2ac36a98cf4c989da P 555fe426f3e5471ba94d6e86c80bda01 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:28:53.241535 31458 raft_consensus.cc:686] T aae5a108c9e543e2ac36a98cf4c989da P 555fe426f3e5471ba94d6e86c80bda01 [term 1 LEADER]: Becoming Leader. State: Replica: 555fe426f3e5471ba94d6e86c80bda01, State: Running, Role: LEADER
I20260928 06:28:53.241614 31458 consensus_queue.cc:237] T aae5a108c9e543e2ac36a98cf4c989da P 555fe426f3e5471ba94d6e86c80bda01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555fe426f3e5471ba94d6e86c80bda01" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 39737 } } peers { permanent_uuid: "1b186b7e369049d9a87566312419b111" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41181 } } peers { permanent_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 44361 } }
I20260928 06:28:53.242251 30941 catalog_manager.cc:5495] T aae5a108c9e543e2ac36a98cf4c989da P 555fe426f3e5471ba94d6e86c80bda01 reported cstate change: term changed from 0 to 1, leader changed from <none> to 555fe426f3e5471ba94d6e86c80bda01 (127.22.225.129). New cstate: current_term: 1 leader_uuid: "555fe426f3e5471ba94d6e86c80bda01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555fe426f3e5471ba94d6e86c80bda01" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 39737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b186b7e369049d9a87566312419b111" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 44361 } health_report { overall_health: UNKNOWN } } }
Sep 28 06:28:53 dist-test-slave-d59m krb5kdc[30897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576932, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.225.190@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM
I20260928 06:28:53.287176 30991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:53.273407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53026 (local address 127.22.225.190:44991)
0928 06:28:53.273456 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:53.273458 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:53.273611 (+ 153us) server_negotiation.cc:407] Connection header received
0928 06:28:53.273702 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:53.273702 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:53.273713 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:53.273786 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:53.274241 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:53.274798 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:53.275554 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:53.275779 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:53.275845 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:53.284313 (+ 8468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:53.284314 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:53.284315 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:53.284322 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:53.285787 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:53.286231 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:53.286233 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:53.286233 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:53.286290 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:53.286481 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:53.286482 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:53.286483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:53.286601 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:53.286761 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:53.286936 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 06:28:53.287031 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 06:28:53.287257 31491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:53.273381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:44991 (local address 127.0.0.1:53026)
0928 06:28:53.273529 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:53.273545 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:53.273628 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:53.273790 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:53.273791 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:53.274113 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:53.274156 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:53.274158 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:53.274884 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:53.274885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:53.275452 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:53.275454 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:53.275871 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:53.275873 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:53.275926 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:53.282737 (+ 6811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:53.282742 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:53.284219 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:53.285924 (+ 1705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:53.285932 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:53.285935 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:53.286141 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:53.286357 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:53.286358 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:53.286359 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:53.286418 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:53.286750 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:53.286758 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:53.286823 (+ 65us) client_negotiation.cc:769] Sending connection context
0928 06:28:53.286913 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 06:28:53.287119 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":81,"thread_start_us":79,"threads_started":1}
I20260928 06:28:53.289691 30941 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.22.225.190@KRBTEST.COM'} at 127.0.0.1:53026:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 06:28:53.289915 30941 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:28:53.291375 31079 tablet_service.cc:1467] Processing CreateTablet for tablet f6ea2103444c4382bbbd157b63808ee4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b25dc71df38f47589cbf6f13326ff8c4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 06:28:53.291376 31219 tablet_service.cc:1467] Processing CreateTablet for tablet f6ea2103444c4382bbbd157b63808ee4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b25dc71df38f47589cbf6f13326ff8c4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 06:28:53.291532 31079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6ea2103444c4382bbbd157b63808ee4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:53.291532 31219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6ea2103444c4382bbbd157b63808ee4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:53.293066 31453 tablet_bootstrap.cc:492] T f6ea2103444c4382bbbd157b63808ee4 P 1b186b7e369049d9a87566312419b111: Bootstrap starting.
I20260928 06:28:53.293148 31455 tablet_bootstrap.cc:492] T f6ea2103444c4382bbbd157b63808ee4 P 555fe426f3e5471ba94d6e86c80bda01: Bootstrap starting.
I20260928 06:28:53.293219 31359 tablet_service.cc:1467] Processing CreateTablet for tablet f6ea2103444c4382bbbd157b63808ee4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b25dc71df38f47589cbf6f13326ff8c4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 06:28:53.293359 31359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6ea2103444c4382bbbd157b63808ee4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:53.293484 31453 tablet_bootstrap.cc:654] T f6ea2103444c4382bbbd157b63808ee4 P 1b186b7e369049d9a87566312419b111: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:53.293531 31455 tablet_bootstrap.cc:654] T f6ea2103444c4382bbbd157b63808ee4 P 555fe426f3e5471ba94d6e86c80bda01: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:53.294426 31453 tablet_bootstrap.cc:492] T f6ea2103444c4382bbbd157b63808ee4 P 1b186b7e369049d9a87566312419b111: No bootstrap required, opened a new log
I20260928 06:28:53.294507 31453 ts_tablet_manager.cc:1397] T f6ea2103444c4382bbbd157b63808ee4 P 1b186b7e369049d9a87566312419b111: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:28:53.294651 31453 raft_consensus.cc:348] T f6ea2103444c4382bbbd157b63808ee4 P 1b186b7e369049d9a87566312419b111 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555fe426f3e5471ba94d6e86c80bda01" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 39737 } } peers { permanent_uuid: "1b186b7e369049d9a87566312419b111" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41181 } } peers { permanent_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 44361 } }
I20260928 06:28:53.294732 31453 raft_consensus.cc:374] T f6ea2103444c4382bbbd157b63808ee4 P 1b186b7e369049d9a87566312419b111 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:53.294759 31453 raft_consensus.cc:729] T f6ea2103444c4382bbbd157b63808ee4 P 1b186b7e369049d9a87566312419b111 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b186b7e369049d9a87566312419b111, State: Initialized, Role: FOLLOWER
I20260928 06:28:53.294787 31457 tablet_bootstrap.cc:492] T f6ea2103444c4382bbbd157b63808ee4 P 6569a7bc6c3a4ab9819f125bcf4fbffd: Bootstrap starting.
I20260928 06:28:53.294801 31453 consensus_queue.cc:260] T f6ea2103444c4382bbbd157b63808ee4 P 1b186b7e369049d9a87566312419b111 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555fe426f3e5471ba94d6e86c80bda01" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 39737 } } peers { permanent_uuid: "1b186b7e369049d9a87566312419b111" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41181 } } peers { permanent_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 44361 } }
I20260928 06:28:53.294885 31453 ts_tablet_manager.cc:1428] T f6ea2103444c4382bbbd157b63808ee4 P 1b186b7e369049d9a87566312419b111: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 06:28:53.294968 31459 tablet_replica.cc:439] T f6ea2103444c4382bbbd157b63808ee4 P 1b186b7e369049d9a87566312419b111: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555fe426f3e5471ba94d6e86c80bda01" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 39737 } } peers { permanent_uuid: "1b186b7e369049d9a87566312419b111" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41181 } } peers { permanent_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 44361 } } }
I20260928 06:28:53.295032 31459 tablet_replica.cc:442] T f6ea2103444c4382bbbd157b63808ee4 P 1b186b7e369049d9a87566312419b111: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:28:53.295194 31457 tablet_bootstrap.cc:654] T f6ea2103444c4382bbbd157b63808ee4 P 6569a7bc6c3a4ab9819f125bcf4fbffd: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:53.295943 31455 tablet_bootstrap.cc:492] T f6ea2103444c4382bbbd157b63808ee4 P 555fe426f3e5471ba94d6e86c80bda01: No bootstrap required, opened a new log
I20260928 06:28:53.295984 31457 tablet_bootstrap.cc:492] T f6ea2103444c4382bbbd157b63808ee4 P 6569a7bc6c3a4ab9819f125bcf4fbffd: No bootstrap required, opened a new log
I20260928 06:28:53.296026 31457 ts_tablet_manager.cc:1397] T f6ea2103444c4382bbbd157b63808ee4 P 6569a7bc6c3a4ab9819f125bcf4fbffd: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:28:53.296007 31455 ts_tablet_manager.cc:1397] T f6ea2103444c4382bbbd157b63808ee4 P 555fe426f3e5471ba94d6e86c80bda01: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 06:28:53.296176 31457 raft_consensus.cc:348] T f6ea2103444c4382bbbd157b63808ee4 P 6569a7bc6c3a4ab9819f125bcf4fbffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555fe426f3e5471ba94d6e86c80bda01" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 39737 } } peers { permanent_uuid: "1b186b7e369049d9a87566312419b111" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41181 } } peers { permanent_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 44361 } }
I20260928 06:28:53.296185 31455 raft_consensus.cc:348] T f6ea2103444c4382bbbd157b63808ee4 P 555fe426f3e5471ba94d6e86c80bda01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555fe426f3e5471ba94d6e86c80bda01" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 39737 } } peers { permanent_uuid: "1b186b7e369049d9a87566312419b111" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41181 } } peers { permanent_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 44361 } }
I20260928 06:28:53.296240 31457 raft_consensus.cc:374] T f6ea2103444c4382bbbd157b63808ee4 P 6569a7bc6c3a4ab9819f125bcf4fbffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:53.296259 31457 raft_consensus.cc:729] T f6ea2103444c4382bbbd157b63808ee4 P 6569a7bc6c3a4ab9819f125bcf4fbffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6569a7bc6c3a4ab9819f125bcf4fbffd, State: Initialized, Role: FOLLOWER
I20260928 06:28:53.296258 31455 raft_consensus.cc:374] T f6ea2103444c4382bbbd157b63808ee4 P 555fe426f3e5471ba94d6e86c80bda01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:53.296285 31455 raft_consensus.cc:729] T f6ea2103444c4382bbbd157b63808ee4 P 555fe426f3e5471ba94d6e86c80bda01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 555fe426f3e5471ba94d6e86c80bda01, State: Initialized, Role: FOLLOWER
I20260928 06:28:53.296296 31457 consensus_queue.cc:260] T f6ea2103444c4382bbbd157b63808ee4 P 6569a7bc6c3a4ab9819f125bcf4fbffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555fe426f3e5471ba94d6e86c80bda01" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 39737 } } peers { permanent_uuid: "1b186b7e369049d9a87566312419b111" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41181 } } peers { permanent_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 44361 } }
I20260928 06:28:53.296329 31455 consensus_queue.cc:260] T f6ea2103444c4382bbbd157b63808ee4 P 555fe426f3e5471ba94d6e86c80bda01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555fe426f3e5471ba94d6e86c80bda01" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 39737 } } peers { permanent_uuid: "1b186b7e369049d9a87566312419b111" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41181 } } peers { permanent_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 44361 } }
I20260928 06:28:53.296367 31457 ts_tablet_manager.cc:1428] T f6ea2103444c4382bbbd157b63808ee4 P 6569a7bc6c3a4ab9819f125bcf4fbffd: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 06:28:53.296428 31455 ts_tablet_manager.cc:1428] T f6ea2103444c4382bbbd157b63808ee4 P 555fe426f3e5471ba94d6e86c80bda01: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 06:28:53.296487 31461 tablet_replica.cc:439] T f6ea2103444c4382bbbd157b63808ee4 P 6569a7bc6c3a4ab9819f125bcf4fbffd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555fe426f3e5471ba94d6e86c80bda01" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 39737 } } peers { permanent_uuid: "1b186b7e369049d9a87566312419b111" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41181 } } peers { permanent_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 44361 } } }
I20260928 06:28:53.296557 31461 tablet_replica.cc:442] T f6ea2103444c4382bbbd157b63808ee4 P 6569a7bc6c3a4ab9819f125bcf4fbffd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:28:53.296581 31458 tablet_replica.cc:439] T f6ea2103444c4382bbbd157b63808ee4 P 555fe426f3e5471ba94d6e86c80bda01: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555fe426f3e5471ba94d6e86c80bda01" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 39737 } } peers { permanent_uuid: "1b186b7e369049d9a87566312419b111" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41181 } } peers { permanent_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 44361 } } }
I20260928 06:28:53.296676 31458 tablet_replica.cc:442] T f6ea2103444c4382bbbd157b63808ee4 P 555fe426f3e5471ba94d6e86c80bda01: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:28:53.485926 31459 raft_consensus.cc:482] T f6ea2103444c4382bbbd157b63808ee4 P 1b186b7e369049d9a87566312419b111 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:28:53.486049 31459 raft_consensus.cc:504] T f6ea2103444c4382bbbd157b63808ee4 P 1b186b7e369049d9a87566312419b111 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555fe426f3e5471ba94d6e86c80bda01" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 39737 } } peers { permanent_uuid: "1b186b7e369049d9a87566312419b111" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41181 } } peers { permanent_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 44361 } }
I20260928 06:28:53.486253 31459 leader_election.cc:288] T f6ea2103444c4382bbbd157b63808ee4 P 1b186b7e369049d9a87566312419b111 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 555fe426f3e5471ba94d6e86c80bda01 (127.22.225.129:39737), 6569a7bc6c3a4ab9819f125bcf4fbffd (127.22.225.131:44361)
I20260928 06:28:53.486449 31099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ea2103444c4382bbbd157b63808ee4" candidate_uuid: "1b186b7e369049d9a87566312419b111" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "555fe426f3e5471ba94d6e86c80bda01" is_pre_election: true
I20260928 06:28:53.486531 31379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ea2103444c4382bbbd157b63808ee4" candidate_uuid: "1b186b7e369049d9a87566312419b111" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" is_pre_election: true
I20260928 06:28:53.486580 31099 raft_consensus.cc:2454] T f6ea2103444c4382bbbd157b63808ee4 P 555fe426f3e5471ba94d6e86c80bda01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b186b7e369049d9a87566312419b111 in term 0.
I20260928 06:28:53.486615 31379 raft_consensus.cc:2454] T f6ea2103444c4382bbbd157b63808ee4 P 6569a7bc6c3a4ab9819f125bcf4fbffd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b186b7e369049d9a87566312419b111 in term 0.
I20260928 06:28:53.486745 31171 leader_election.cc:302] T f6ea2103444c4382bbbd157b63808ee4 P 1b186b7e369049d9a87566312419b111 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b186b7e369049d9a87566312419b111, 555fe426f3e5471ba94d6e86c80bda01; no voters:
I20260928 06:28:53.486882 31459 raft_consensus.cc:2781] T f6ea2103444c4382bbbd157b63808ee4 P 1b186b7e369049d9a87566312419b111 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:28:53.486953 31459 raft_consensus.cc:482] T f6ea2103444c4382bbbd157b63808ee4 P 1b186b7e369049d9a87566312419b111 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:28:53.486979 31459 raft_consensus.cc:3037] T f6ea2103444c4382bbbd157b63808ee4 P 1b186b7e369049d9a87566312419b111 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:53.487887 31459 raft_consensus.cc:504] T f6ea2103444c4382bbbd157b63808ee4 P 1b186b7e369049d9a87566312419b111 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555fe426f3e5471ba94d6e86c80bda01" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 39737 } } peers { permanent_uuid: "1b186b7e369049d9a87566312419b111" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41181 } } peers { permanent_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 44361 } }
I20260928 06:28:53.488034 31459 leader_election.cc:288] T f6ea2103444c4382bbbd157b63808ee4 P 1b186b7e369049d9a87566312419b111 [CANDIDATE]: Term 1 election: Requested vote from peers 555fe426f3e5471ba94d6e86c80bda01 (127.22.225.129:39737), 6569a7bc6c3a4ab9819f125bcf4fbffd (127.22.225.131:44361)
I20260928 06:28:53.488186 31379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ea2103444c4382bbbd157b63808ee4" candidate_uuid: "1b186b7e369049d9a87566312419b111" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd"
I20260928 06:28:53.488241 31099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ea2103444c4382bbbd157b63808ee4" candidate_uuid: "1b186b7e369049d9a87566312419b111" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "555fe426f3e5471ba94d6e86c80bda01"
I20260928 06:28:53.488289 31379 raft_consensus.cc:3037] T f6ea2103444c4382bbbd157b63808ee4 P 6569a7bc6c3a4ab9819f125bcf4fbffd [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:53.488328 31099 raft_consensus.cc:3037] T f6ea2103444c4382bbbd157b63808ee4 P 555fe426f3e5471ba94d6e86c80bda01 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:53.488960 31379 raft_consensus.cc:2454] T f6ea2103444c4382bbbd157b63808ee4 P 6569a7bc6c3a4ab9819f125bcf4fbffd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b186b7e369049d9a87566312419b111 in term 1.
I20260928 06:28:53.489146 31099 raft_consensus.cc:2454] T f6ea2103444c4382bbbd157b63808ee4 P 555fe426f3e5471ba94d6e86c80bda01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b186b7e369049d9a87566312419b111 in term 1.
I20260928 06:28:53.489146 31173 leader_election.cc:302] T f6ea2103444c4382bbbd157b63808ee4 P 1b186b7e369049d9a87566312419b111 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b186b7e369049d9a87566312419b111, 6569a7bc6c3a4ab9819f125bcf4fbffd; no voters:
I20260928 06:28:53.489329 31459 raft_consensus.cc:2781] T f6ea2103444c4382bbbd157b63808ee4 P 1b186b7e369049d9a87566312419b111 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:28:53.489382 31459 raft_consensus.cc:686] T f6ea2103444c4382bbbd157b63808ee4 P 1b186b7e369049d9a87566312419b111 [term 1 LEADER]: Becoming Leader. State: Replica: 1b186b7e369049d9a87566312419b111, State: Running, Role: LEADER
I20260928 06:28:53.489439 31459 consensus_queue.cc:237] T f6ea2103444c4382bbbd157b63808ee4 P 1b186b7e369049d9a87566312419b111 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555fe426f3e5471ba94d6e86c80bda01" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 39737 } } peers { permanent_uuid: "1b186b7e369049d9a87566312419b111" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41181 } } peers { permanent_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 44361 } }
I20260928 06:28:53.489667 31463 tablet_replica.cc:439] T f6ea2103444c4382bbbd157b63808ee4 P 1b186b7e369049d9a87566312419b111: TxnStatusTablet state changed. Reason: New leader 1b186b7e369049d9a87566312419b111. Latest consensus state: current_term: 1 leader_uuid: "1b186b7e369049d9a87566312419b111" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555fe426f3e5471ba94d6e86c80bda01" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 39737 } } peers { permanent_uuid: "1b186b7e369049d9a87566312419b111" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41181 } } peers { permanent_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 44361 } } }
I20260928 06:28:53.489743 31463 tablet_replica.cc:442] T f6ea2103444c4382bbbd157b63808ee4 P 1b186b7e369049d9a87566312419b111: This TxnStatusTablet replica's current role is: LEADER
I20260928 06:28:53.489887 31496 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 06:28:53.490065 31496 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 06:28:53.490137 30941 catalog_manager.cc:5495] T f6ea2103444c4382bbbd157b63808ee4 P 1b186b7e369049d9a87566312419b111 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b186b7e369049d9a87566312419b111 (127.22.225.130). New cstate: current_term: 1 leader_uuid: "1b186b7e369049d9a87566312419b111" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555fe426f3e5471ba94d6e86c80bda01" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 39737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b186b7e369049d9a87566312419b111" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 44361 } health_report { overall_health: UNKNOWN } } }
I20260928 06:28:53.527724 31451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:53.524556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57150 (local address 127.22.225.130:41181)
0928 06:28:53.524622 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:53.524624 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:53.524634 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:28:53.524740 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:53.524741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:53.524751 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:53.524817 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:28:53.524962 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:53.525740 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:53.526882 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:53.527193 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:53.527265 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:53.527464 (+ 199us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:28:53.527501 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:28:53.527550 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:53.527621 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 06:28:53.527640 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 06:28:53.527755 31491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:53.524486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.130:41181 (local address 127.0.0.1:57150)
0928 06:28:53.524591 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:53.524602 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:53.524647 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:53.524837 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:53.524838 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:53.524840 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:28:53.524891 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:53.524893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:53.525868 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:53.525870 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:53.526792 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:53.526794 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:53.527275 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:53.527276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:53.527349 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:53.527366 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:28:53.527565 (+ 199us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:28:53.527572 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 06:28:53.527621 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 06:28:53.527672 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 06:28:53.528745 31099 raft_consensus.cc:1261] T f6ea2103444c4382bbbd157b63808ee4 P 555fe426f3e5471ba94d6e86c80bda01 [term 1 FOLLOWER]: Refusing update from remote peer 1b186b7e369049d9a87566312419b111: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:28:53.528771 31379 raft_consensus.cc:1261] T f6ea2103444c4382bbbd157b63808ee4 P 6569a7bc6c3a4ab9819f125bcf4fbffd [term 1 FOLLOWER]: Refusing update from remote peer 1b186b7e369049d9a87566312419b111: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:28:53.528990 31459 consensus_queue.cc:1035] T f6ea2103444c4382bbbd157b63808ee4 P 1b186b7e369049d9a87566312419b111 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 44361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:28:53.529114 31459 consensus_queue.cc:1035] T f6ea2103444c4382bbbd157b63808ee4 P 1b186b7e369049d9a87566312419b111 [LEADER]: Connected to new peer: Peer: permanent_uuid: "555fe426f3e5471ba94d6e86c80bda01" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 39737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:28:53.529596 31461 tablet_replica.cc:439] T f6ea2103444c4382bbbd157b63808ee4 P 6569a7bc6c3a4ab9819f125bcf4fbffd: TxnStatusTablet state changed. Reason: New leader 1b186b7e369049d9a87566312419b111. Latest consensus state: current_term: 1 leader_uuid: "1b186b7e369049d9a87566312419b111" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555fe426f3e5471ba94d6e86c80bda01" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 39737 } } peers { permanent_uuid: "1b186b7e369049d9a87566312419b111" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41181 } } peers { permanent_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 44361 } } }
I20260928 06:28:53.529608 31458 tablet_replica.cc:439] T f6ea2103444c4382bbbd157b63808ee4 P 555fe426f3e5471ba94d6e86c80bda01: TxnStatusTablet state changed. Reason: New leader 1b186b7e369049d9a87566312419b111. Latest consensus state: current_term: 1 leader_uuid: "1b186b7e369049d9a87566312419b111" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555fe426f3e5471ba94d6e86c80bda01" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 39737 } } peers { permanent_uuid: "1b186b7e369049d9a87566312419b111" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41181 } } peers { permanent_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 44361 } } }
I20260928 06:28:53.529690 31461 tablet_replica.cc:442] T f6ea2103444c4382bbbd157b63808ee4 P 6569a7bc6c3a4ab9819f125bcf4fbffd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:28:53.529702 31458 tablet_replica.cc:442] T f6ea2103444c4382bbbd157b63808ee4 P 555fe426f3e5471ba94d6e86c80bda01: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:28:53.530185 31463 tablet_replica.cc:439] T f6ea2103444c4382bbbd157b63808ee4 P 1b186b7e369049d9a87566312419b111: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1b186b7e369049d9a87566312419b111" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555fe426f3e5471ba94d6e86c80bda01" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 39737 } } peers { permanent_uuid: "1b186b7e369049d9a87566312419b111" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41181 } } peers { permanent_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 44361 } } }
I20260928 06:28:53.530260 31463 tablet_replica.cc:442] T f6ea2103444c4382bbbd157b63808ee4 P 1b186b7e369049d9a87566312419b111: This TxnStatusTablet replica's current role is: LEADER
I20260928 06:28:53.530604 31466 mvcc.cc:205] Tried to move back new op lower bound from 7334203119731875840 to 7334203119573127168. Current Snapshot: MvccSnapshot[applied={T|T < 7334203119731875840 or (T in {7334203119731875840})}]
I20260928 06:28:53.531044 31468 mvcc.cc:205] Tried to move back new op lower bound from 7334203119731875840 to 7334203119573127168. Current Snapshot: MvccSnapshot[applied={T|T < 7334203119731875840}]
I20260928 06:28:53.531136 31461 tablet_replica.cc:439] T f6ea2103444c4382bbbd157b63808ee4 P 6569a7bc6c3a4ab9819f125bcf4fbffd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1b186b7e369049d9a87566312419b111" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555fe426f3e5471ba94d6e86c80bda01" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 39737 } } peers { permanent_uuid: "1b186b7e369049d9a87566312419b111" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41181 } } peers { permanent_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 44361 } } }
I20260928 06:28:53.531194 31461 tablet_replica.cc:442] T f6ea2103444c4382bbbd157b63808ee4 P 6569a7bc6c3a4ab9819f125bcf4fbffd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:28:53.532342 31459 tablet_replica.cc:439] T f6ea2103444c4382bbbd157b63808ee4 P 1b186b7e369049d9a87566312419b111: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1b186b7e369049d9a87566312419b111" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555fe426f3e5471ba94d6e86c80bda01" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 39737 } } peers { permanent_uuid: "1b186b7e369049d9a87566312419b111" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41181 } } peers { permanent_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 44361 } } }
I20260928 06:28:53.532413 31459 tablet_replica.cc:442] T f6ea2103444c4382bbbd157b63808ee4 P 1b186b7e369049d9a87566312419b111: This TxnStatusTablet replica's current role is: LEADER
I20260928 06:28:53.533005 31467 mvcc.cc:205] Tried to move back new op lower bound from 7334203119731875840 to 7334203119573127168. Current Snapshot: MvccSnapshot[applied={T|T < 7334203119731875840}]
I20260928 06:28:53.533376 31458 tablet_replica.cc:439] T f6ea2103444c4382bbbd157b63808ee4 P 555fe426f3e5471ba94d6e86c80bda01: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1b186b7e369049d9a87566312419b111" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555fe426f3e5471ba94d6e86c80bda01" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 39737 } } peers { permanent_uuid: "1b186b7e369049d9a87566312419b111" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41181 } } peers { permanent_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 44361 } } }
I20260928 06:28:53.533452 31458 tablet_replica.cc:442] T f6ea2103444c4382bbbd157b63808ee4 P 555fe426f3e5471ba94d6e86c80bda01: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:28:53.535778 31450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:53.532114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55470 (local address 127.22.225.129:39737)
0928 06:28:53.532151 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:53.532154 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:53.532161 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:28:53.532197 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:53.532198 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:53.532216 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:53.532282 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:28:53.532493 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:53.533265 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:53.534675 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:53.534974 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:53.535032 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:53.535226 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:28:53.535372 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:28:53.535449 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:53.535686 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 06:28:53.535704 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 06:28:53.541472 31451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:53.538461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:55789 (local address 127.22.225.130:41181)
0928 06:28:53.538500 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:53.538502 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:53.538508 (+ 6us) server_negotiation.cc:407] Connection header received
0928 06:28:53.538527 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:53.538528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:53.538536 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:53.538580 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:28:53.538746 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:53.539467 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:53.540668 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:53.540962 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:53.541000 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:53.541213 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:28:53.541243 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:28:53.541268 (+ 25us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:53.541388 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 06:28:53.541405 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:28:53.541826 31503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:53.538233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.130:41181 (local address 127.22.225.129:55789)
0928 06:28:53.538380 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:53.538390 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:53.538440 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:53.538616 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:53.538617 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:53.538618 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:28:53.538678 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:53.538679 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:53.539579 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:53.539580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:53.540531 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:53.540533 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:53.541053 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:53.541053 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:53.541118 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:53.541131 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:28:53.541328 (+ 197us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:28:53.541329 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 06:28:53.541675 (+ 346us) client_negotiation.cc:240] Negotiation successful
0928 06:28:53.541716 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":83,"threads_started":1}
I20260928 06:28:53.544540 31379 raft_consensus.cc:1261] T aae5a108c9e543e2ac36a98cf4c989da P 6569a7bc6c3a4ab9819f125bcf4fbffd [term 1 FOLLOWER]: Refusing update from remote peer 555fe426f3e5471ba94d6e86c80bda01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:28:53.544647 31239 raft_consensus.cc:1261] T aae5a108c9e543e2ac36a98cf4c989da P 1b186b7e369049d9a87566312419b111 [term 1 FOLLOWER]: Refusing update from remote peer 555fe426f3e5471ba94d6e86c80bda01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:28:53.544791 31484 consensus_queue.cc:1035] T aae5a108c9e543e2ac36a98cf4c989da P 555fe426f3e5471ba94d6e86c80bda01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 44361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:28:53.545065 31484 consensus_queue.cc:1035] T aae5a108c9e543e2ac36a98cf4c989da P 555fe426f3e5471ba94d6e86c80bda01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b186b7e369049d9a87566312419b111" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:28:53.555409 31450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:53.551562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:43729 (local address 127.22.225.129:39737)
0928 06:28:53.551601 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:53.551604 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:53.551711 (+ 107us) server_negotiation.cc:407] Connection header received
0928 06:28:53.551775 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:53.551776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:53.551788 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:53.551847 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:28:53.552014 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:53.552782 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:53.554575 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:53.554879 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:53.554923 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:53.555135 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:28:53.555167 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:28:53.555194 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:53.555290 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 06:28:53.555322 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:28:53.556272 31507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:53.551469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.129:39737 (local address 127.22.225.130:43729)
0928 06:28:53.551653 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:53.551664 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:53.551722 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:53.551891 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:53.551892 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:53.551894 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:28:53.551938 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:53.551940 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:53.552876 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:53.552877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:53.554463 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:53.554467 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:53.554982 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:53.554982 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:53.555039 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:53.555053 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:28:53.555231 (+ 178us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:28:53.555232 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 06:28:53.556148 (+ 916us) client_negotiation.cc:240] Negotiation successful
0928 06:28:53.556193 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":81,"threads_started":1}
I20260928 06:28:53.571735 30941 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:28:53.571837 30941 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:28:53.572407 30941 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 58908c52510e473e923860f490430124: Sending DeleteTablet for 3 replicas of tablet aae5a108c9e543e2ac36a98cf4c989da
I20260928 06:28:53.572726 31359 tablet_service.cc:1514] Processing DeleteTablet for tablet aae5a108c9e543e2ac36a98cf4c989da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:28:53 UTC) from {username='oryx', principal='oryx/127.22.225.190@KRBTEST.COM'} at 127.0.0.1:35664
I20260928 06:28:53.572749 31079 tablet_service.cc:1514] Processing DeleteTablet for tablet aae5a108c9e543e2ac36a98cf4c989da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:28:53 UTC) from {username='oryx', principal='oryx/127.22.225.190@KRBTEST.COM'} at 127.0.0.1:55466
I20260928 06:28:53.572830 31476 tablet_replica.cc:330] T aae5a108c9e543e2ac36a98cf4c989da P 6569a7bc6c3a4ab9819f125bcf4fbffd: stopping tablet replica
I20260928 06:28:53.572880 31476 raft_consensus.cc:2229] T aae5a108c9e543e2ac36a98cf4c989da P 6569a7bc6c3a4ab9819f125bcf4fbffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:28:53.572911 31476 raft_consensus.cc:2258] T aae5a108c9e543e2ac36a98cf4c989da P 6569a7bc6c3a4ab9819f125bcf4fbffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:28:53.572968 31477 tablet_replica.cc:330] T aae5a108c9e543e2ac36a98cf4c989da P 555fe426f3e5471ba94d6e86c80bda01: stopping tablet replica
I20260928 06:28:53.573019 31477 raft_consensus.cc:2229] T aae5a108c9e543e2ac36a98cf4c989da P 555fe426f3e5471ba94d6e86c80bda01 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:28:53.573102 31477 raft_consensus.cc:2258] T aae5a108c9e543e2ac36a98cf4c989da P 555fe426f3e5471ba94d6e86c80bda01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:28:53.573489 31476 ts_tablet_manager.cc:1905] T aae5a108c9e543e2ac36a98cf4c989da P 6569a7bc6c3a4ab9819f125bcf4fbffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:28:53.575309 31476 ts_tablet_manager.cc:1918] T aae5a108c9e543e2ac36a98cf4c989da P 6569a7bc6c3a4ab9819f125bcf4fbffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:28:53.575371 31476 log.cc:1196] T aae5a108c9e543e2ac36a98cf4c989da P 6569a7bc6c3a4ab9819f125bcf4fbffd: Deleting WAL directory at /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/ts-2/wal/wals/aae5a108c9e543e2ac36a98cf4c989da
I20260928 06:28:53.575588 31476 ts_tablet_manager.cc:1939] T aae5a108c9e543e2ac36a98cf4c989da P 6569a7bc6c3a4ab9819f125bcf4fbffd: Deleting consensus metadata
I20260928 06:28:53.575938 31477 ts_tablet_manager.cc:1905] T aae5a108c9e543e2ac36a98cf4c989da P 555fe426f3e5471ba94d6e86c80bda01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:28:53.576851 31219 tablet_service.cc:1514] Processing DeleteTablet for tablet aae5a108c9e543e2ac36a98cf4c989da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:28:53 UTC) from {username='oryx', principal='oryx/127.22.225.190@KRBTEST.COM'} at 127.0.0.1:57146
I20260928 06:28:53.576972 31478 tablet_replica.cc:330] T aae5a108c9e543e2ac36a98cf4c989da P 1b186b7e369049d9a87566312419b111: stopping tablet replica
I20260928 06:28:53.577049 31478 raft_consensus.cc:2229] T aae5a108c9e543e2ac36a98cf4c989da P 1b186b7e369049d9a87566312419b111 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:28:53.577087 31478 raft_consensus.cc:2258] T aae5a108c9e543e2ac36a98cf4c989da P 1b186b7e369049d9a87566312419b111 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:28:53.577677 31477 ts_tablet_manager.cc:1918] T aae5a108c9e543e2ac36a98cf4c989da P 555fe426f3e5471ba94d6e86c80bda01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:28:53.577754 31477 log.cc:1196] T aae5a108c9e543e2ac36a98cf4c989da P 555fe426f3e5471ba94d6e86c80bda01: Deleting WAL directory at /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/ts-0/wal/wals/aae5a108c9e543e2ac36a98cf4c989da
I20260928 06:28:53.577991 31477 ts_tablet_manager.cc:1939] T aae5a108c9e543e2ac36a98cf4c989da P 555fe426f3e5471ba94d6e86c80bda01: Deleting consensus metadata
I20260928 06:28:53.578461 31478 ts_tablet_manager.cc:1905] T aae5a108c9e543e2ac36a98cf4c989da P 1b186b7e369049d9a87566312419b111: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:28:53.578464 30927 catalog_manager.cc:4841] TS 555fe426f3e5471ba94d6e86c80bda01 (127.22.225.129:39737): tablet aae5a108c9e543e2ac36a98cf4c989da (table test-table [id=ee810257885c4240bfd23fd8e844828e]) successfully deleted
I20260928 06:28:53.578557 30929 catalog_manager.cc:4841] TS 6569a7bc6c3a4ab9819f125bcf4fbffd (127.22.225.131:44361): tablet aae5a108c9e543e2ac36a98cf4c989da (table test-table [id=ee810257885c4240bfd23fd8e844828e]) successfully deleted
I20260928 06:28:53.580005 31478 ts_tablet_manager.cc:1918] T aae5a108c9e543e2ac36a98cf4c989da P 1b186b7e369049d9a87566312419b111: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:28:53.580063 31478 log.cc:1196] T aae5a108c9e543e2ac36a98cf4c989da P 1b186b7e369049d9a87566312419b111: Deleting WAL directory at /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/ts-1/wal/wals/aae5a108c9e543e2ac36a98cf4c989da
I20260928 06:28:53.580282 31478 ts_tablet_manager.cc:1939] T aae5a108c9e543e2ac36a98cf4c989da P 1b186b7e369049d9a87566312419b111: Deleting consensus metadata
I20260928 06:28:53.580757 30927 catalog_manager.cc:4841] TS 1b186b7e369049d9a87566312419b111 (127.22.225.130:41181): tablet aae5a108c9e543e2ac36a98cf4c989da (table test-table [id=ee810257885c4240bfd23fd8e844828e]) successfully deleted
Sep 28 06:28:53 dist-test-slave-d59m krb5kdc[30897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 06:28:53 dist-test-slave-d59m krb5kdc[30897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:28:53 dist-test-slave-d59m krb5kdc[30897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
W20260928 06:28:53.610715 30991 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:28:53.603452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53036 (local address 127.22.225.190:44991)
0928 06:28:53.603501 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:53.603503 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:53.603642 (+ 139us) server_negotiation.cc:407] Connection header received
0928 06:28:53.603758 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:53.603759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:53.603771 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:53.603803 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:53.604557 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:53.605311 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:53.606623 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:53.606982 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:53.607049 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:53.610645 (+ 3596us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53036: BlockingRecv error: recv got EOF from 127.0.0.1:53036 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 28 06:28:53 dist-test-slave-d59m krb5kdc[30897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM
I20260928 06:28:53.620838 30991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:53.611886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53042 (local address 127.22.225.190:44991)
0928 06:28:53.611929 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:53.611931 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:53.612258 (+ 327us) server_negotiation.cc:407] Connection header received
0928 06:28:53.612354 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:53.612354 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:53.612365 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:53.612393 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:53.613083 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:53.613759 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:53.614703 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:53.614996 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:53.615040 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:53.617840 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:53.617842 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:53.617842 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:53.617849 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:53.619512 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:53.619913 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:53.619914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:53.619914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:53.619981 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:53.620190 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:53.620191 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:53.620191 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:53.620320 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:53.620394 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:53.620656 (+ 262us) server_negotiation.cc:299] Negotiation successful
0928 06:28:53.620733 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 06:28:53.622759 30941 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53042:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:28:53.622943 30941 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:28:53.624276 31359 tablet_service.cc:1467] Processing CreateTablet for tablet ea222aaa3635440d946303b97bbf4f83 (DEFAULT_TABLE table=test-table [id=5db8b12f8c3543b3b8e3e6fbfb0abbd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:28:53.624264 31079 tablet_service.cc:1467] Processing CreateTablet for tablet ea222aaa3635440d946303b97bbf4f83 (DEFAULT_TABLE table=test-table [id=5db8b12f8c3543b3b8e3e6fbfb0abbd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:28:53.624300 31219 tablet_service.cc:1467] Processing CreateTablet for tablet ea222aaa3635440d946303b97bbf4f83 (DEFAULT_TABLE table=test-table [id=5db8b12f8c3543b3b8e3e6fbfb0abbd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:28:53.624406 31359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea222aaa3635440d946303b97bbf4f83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:53.624408 31079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea222aaa3635440d946303b97bbf4f83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:53.624424 31219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea222aaa3635440d946303b97bbf4f83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:53.625730 31453 tablet_bootstrap.cc:492] T ea222aaa3635440d946303b97bbf4f83 P 1b186b7e369049d9a87566312419b111: Bootstrap starting.
I20260928 06:28:53.625846 31455 tablet_bootstrap.cc:492] T ea222aaa3635440d946303b97bbf4f83 P 555fe426f3e5471ba94d6e86c80bda01: Bootstrap starting.
I20260928 06:28:53.625788 31457 tablet_bootstrap.cc:492] T ea222aaa3635440d946303b97bbf4f83 P 6569a7bc6c3a4ab9819f125bcf4fbffd: Bootstrap starting.
I20260928 06:28:53.626272 31457 tablet_bootstrap.cc:654] T ea222aaa3635440d946303b97bbf4f83 P 6569a7bc6c3a4ab9819f125bcf4fbffd: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:53.626272 31455 tablet_bootstrap.cc:654] T ea222aaa3635440d946303b97bbf4f83 P 555fe426f3e5471ba94d6e86c80bda01: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:53.626626 31453 tablet_bootstrap.cc:654] T ea222aaa3635440d946303b97bbf4f83 P 1b186b7e369049d9a87566312419b111: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:53.627365 31453 tablet_bootstrap.cc:492] T ea222aaa3635440d946303b97bbf4f83 P 1b186b7e369049d9a87566312419b111: No bootstrap required, opened a new log
I20260928 06:28:53.627410 31453 ts_tablet_manager.cc:1397] T ea222aaa3635440d946303b97bbf4f83 P 1b186b7e369049d9a87566312419b111: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:28:53.627521 31453 raft_consensus.cc:348] T ea222aaa3635440d946303b97bbf4f83 P 1b186b7e369049d9a87566312419b111 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555fe426f3e5471ba94d6e86c80bda01" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 39737 } } peers { permanent_uuid: "1b186b7e369049d9a87566312419b111" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41181 } } peers { permanent_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 44361 } }
I20260928 06:28:53.627579 31453 raft_consensus.cc:374] T ea222aaa3635440d946303b97bbf4f83 P 1b186b7e369049d9a87566312419b111 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:53.627599 31453 raft_consensus.cc:729] T ea222aaa3635440d946303b97bbf4f83 P 1b186b7e369049d9a87566312419b111 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b186b7e369049d9a87566312419b111, State: Initialized, Role: FOLLOWER
I20260928 06:28:53.627635 31453 consensus_queue.cc:260] T ea222aaa3635440d946303b97bbf4f83 P 1b186b7e369049d9a87566312419b111 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555fe426f3e5471ba94d6e86c80bda01" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 39737 } } peers { permanent_uuid: "1b186b7e369049d9a87566312419b111" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41181 } } peers { permanent_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 44361 } }
I20260928 06:28:53.627717 31453 ts_tablet_manager.cc:1428] T ea222aaa3635440d946303b97bbf4f83 P 1b186b7e369049d9a87566312419b111: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:53.629082 31455 tablet_bootstrap.cc:492] T ea222aaa3635440d946303b97bbf4f83 P 555fe426f3e5471ba94d6e86c80bda01: No bootstrap required, opened a new log
I20260928 06:28:53.629139 31455 ts_tablet_manager.cc:1397] T ea222aaa3635440d946303b97bbf4f83 P 555fe426f3e5471ba94d6e86c80bda01: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 06:28:53.629279 31455 raft_consensus.cc:348] T ea222aaa3635440d946303b97bbf4f83 P 555fe426f3e5471ba94d6e86c80bda01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555fe426f3e5471ba94d6e86c80bda01" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 39737 } } peers { permanent_uuid: "1b186b7e369049d9a87566312419b111" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41181 } } peers { permanent_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 44361 } }
I20260928 06:28:53.629374 31455 raft_consensus.cc:374] T ea222aaa3635440d946303b97bbf4f83 P 555fe426f3e5471ba94d6e86c80bda01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:53.629395 31455 raft_consensus.cc:729] T ea222aaa3635440d946303b97bbf4f83 P 555fe426f3e5471ba94d6e86c80bda01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 555fe426f3e5471ba94d6e86c80bda01, State: Initialized, Role: FOLLOWER
I20260928 06:28:53.629439 31455 consensus_queue.cc:260] T ea222aaa3635440d946303b97bbf4f83 P 555fe426f3e5471ba94d6e86c80bda01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555fe426f3e5471ba94d6e86c80bda01" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 39737 } } peers { permanent_uuid: "1b186b7e369049d9a87566312419b111" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41181 } } peers { permanent_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 44361 } }
I20260928 06:28:53.629521 31455 ts_tablet_manager.cc:1428] T ea222aaa3635440d946303b97bbf4f83 P 555fe426f3e5471ba94d6e86c80bda01: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:53.635632 31457 tablet_bootstrap.cc:492] T ea222aaa3635440d946303b97bbf4f83 P 6569a7bc6c3a4ab9819f125bcf4fbffd: No bootstrap required, opened a new log
I20260928 06:28:53.635682 31457 ts_tablet_manager.cc:1397] T ea222aaa3635440d946303b97bbf4f83 P 6569a7bc6c3a4ab9819f125bcf4fbffd: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20260928 06:28:53.635828 31457 raft_consensus.cc:348] T ea222aaa3635440d946303b97bbf4f83 P 6569a7bc6c3a4ab9819f125bcf4fbffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555fe426f3e5471ba94d6e86c80bda01" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 39737 } } peers { permanent_uuid: "1b186b7e369049d9a87566312419b111" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41181 } } peers { permanent_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 44361 } }
I20260928 06:28:53.635890 31457 raft_consensus.cc:374] T ea222aaa3635440d946303b97bbf4f83 P 6569a7bc6c3a4ab9819f125bcf4fbffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:53.635908 31457 raft_consensus.cc:729] T ea222aaa3635440d946303b97bbf4f83 P 6569a7bc6c3a4ab9819f125bcf4fbffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6569a7bc6c3a4ab9819f125bcf4fbffd, State: Initialized, Role: FOLLOWER
I20260928 06:28:53.635954 31457 consensus_queue.cc:260] T ea222aaa3635440d946303b97bbf4f83 P 6569a7bc6c3a4ab9819f125bcf4fbffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555fe426f3e5471ba94d6e86c80bda01" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 39737 } } peers { permanent_uuid: "1b186b7e369049d9a87566312419b111" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41181 } } peers { permanent_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 44361 } }
I20260928 06:28:53.636022 31457 ts_tablet_manager.cc:1428] T ea222aaa3635440d946303b97bbf4f83 P 6569a7bc6c3a4ab9819f125bcf4fbffd: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:53.836859 31461 raft_consensus.cc:482] T ea222aaa3635440d946303b97bbf4f83 P 6569a7bc6c3a4ab9819f125bcf4fbffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:28:53.837001 31461 raft_consensus.cc:504] T ea222aaa3635440d946303b97bbf4f83 P 6569a7bc6c3a4ab9819f125bcf4fbffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555fe426f3e5471ba94d6e86c80bda01" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 39737 } } peers { permanent_uuid: "1b186b7e369049d9a87566312419b111" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41181 } } peers { permanent_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 44361 } }
I20260928 06:28:53.837378 31461 leader_election.cc:288] T ea222aaa3635440d946303b97bbf4f83 P 6569a7bc6c3a4ab9819f125bcf4fbffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 555fe426f3e5471ba94d6e86c80bda01 (127.22.225.129:39737), 1b186b7e369049d9a87566312419b111 (127.22.225.130:41181)
I20260928 06:28:53.841465 31451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:53.837939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:59675 (local address 127.22.225.130:41181)
0928 06:28:53.837993 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:53.837995 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:53.838007 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:28:53.838061 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:53.838062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:53.838072 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:53.838131 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:28:53.838329 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:53.839049 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:53.840648 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:53.841024 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:53.841111 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:53.841120 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:53.841347 (+ 227us) server_negotiation.cc:299] Negotiation successful
0928 06:28:53.841395 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 06:28:53.841544 31450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:53.837672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:49221 (local address 127.22.225.129:39737)
0928 06:28:53.837725 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:53.837727 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:53.837818 (+ 91us) server_negotiation.cc:407] Connection header received
0928 06:28:53.837938 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:53.837940 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:53.837955 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:53.838007 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:28:53.838223 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:53.838994 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:53.840635 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:53.841158 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:53.841242 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:53.841250 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:53.841420 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 06:28:53.841470 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 06:28:53.841768 31526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:53.837603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.129:39737 (local address 127.22.225.131:49221)
0928 06:28:53.837719 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:53.837731 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:53.837818 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:53.838086 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:53.838087 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:53.838089 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:28:53.838151 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:53.838153 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:53.839140 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:53.839141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:53.840508 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:53.840510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:53.841268 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:53.841270 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:53.841328 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:53.841335 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 06:28:53.841645 (+ 310us) client_negotiation.cc:240] Negotiation successful
0928 06:28:53.841687 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"mutex_wait_us":84,"thread_start_us":62,"threads_started":1}
I20260928 06:28:53.842032 31099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea222aaa3635440d946303b97bbf4f83" candidate_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "555fe426f3e5471ba94d6e86c80bda01" is_pre_election: true
I20260928 06:28:53.842136 31099 raft_consensus.cc:2454] T ea222aaa3635440d946303b97bbf4f83 P 555fe426f3e5471ba94d6e86c80bda01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6569a7bc6c3a4ab9819f125bcf4fbffd in term 0.
I20260928 06:28:53.842128 31527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:53.837774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.130:41181 (local address 127.22.225.131:59675)
0928 06:28:53.837943 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:53.837953 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:53.838004 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:53.838154 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:53.838155 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:53.838157 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:28:53.838242 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:53.838244 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:53.839164 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:53.839164 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:53.840507 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:53.840510 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:53.841112 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:53.841112 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:53.841192 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:53.841200 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 06:28:53.841314 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 06:28:53.841362 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":30,"thread_start_us":43,"threads_started":1}
I20260928 06:28:53.842346 31311 leader_election.cc:302] T ea222aaa3635440d946303b97bbf4f83 P 6569a7bc6c3a4ab9819f125bcf4fbffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 555fe426f3e5471ba94d6e86c80bda01, 6569a7bc6c3a4ab9819f125bcf4fbffd; no voters:
I20260928 06:28:53.842470 31461 raft_consensus.cc:2781] T ea222aaa3635440d946303b97bbf4f83 P 6569a7bc6c3a4ab9819f125bcf4fbffd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:28:53.842526 31461 raft_consensus.cc:482] T ea222aaa3635440d946303b97bbf4f83 P 6569a7bc6c3a4ab9819f125bcf4fbffd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:28:53.842548 31461 raft_consensus.cc:3037] T ea222aaa3635440d946303b97bbf4f83 P 6569a7bc6c3a4ab9819f125bcf4fbffd [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:53.842612 31239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea222aaa3635440d946303b97bbf4f83" candidate_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b186b7e369049d9a87566312419b111" is_pre_election: true
I20260928 06:28:53.842705 31239 raft_consensus.cc:2454] T ea222aaa3635440d946303b97bbf4f83 P 1b186b7e369049d9a87566312419b111 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6569a7bc6c3a4ab9819f125bcf4fbffd in term 0.
I20260928 06:28:53.843509 31461 raft_consensus.cc:504] T ea222aaa3635440d946303b97bbf4f83 P 6569a7bc6c3a4ab9819f125bcf4fbffd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555fe426f3e5471ba94d6e86c80bda01" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 39737 } } peers { permanent_uuid: "1b186b7e369049d9a87566312419b111" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41181 } } peers { permanent_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 44361 } }
I20260928 06:28:53.843607 31461 leader_election.cc:288] T ea222aaa3635440d946303b97bbf4f83 P 6569a7bc6c3a4ab9819f125bcf4fbffd [CANDIDATE]: Term 1 election: Requested vote from peers 555fe426f3e5471ba94d6e86c80bda01 (127.22.225.129:39737), 1b186b7e369049d9a87566312419b111 (127.22.225.130:41181)
I20260928 06:28:53.843789 31099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea222aaa3635440d946303b97bbf4f83" candidate_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "555fe426f3e5471ba94d6e86c80bda01"
I20260928 06:28:53.843807 31239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea222aaa3635440d946303b97bbf4f83" candidate_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b186b7e369049d9a87566312419b111"
I20260928 06:28:53.843888 31099 raft_consensus.cc:3037] T ea222aaa3635440d946303b97bbf4f83 P 555fe426f3e5471ba94d6e86c80bda01 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:53.843886 31239 raft_consensus.cc:3037] T ea222aaa3635440d946303b97bbf4f83 P 1b186b7e369049d9a87566312419b111 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:53.844555 31099 raft_consensus.cc:2454] T ea222aaa3635440d946303b97bbf4f83 P 555fe426f3e5471ba94d6e86c80bda01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6569a7bc6c3a4ab9819f125bcf4fbffd in term 1.
I20260928 06:28:53.844656 31239 raft_consensus.cc:2454] T ea222aaa3635440d946303b97bbf4f83 P 1b186b7e369049d9a87566312419b111 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6569a7bc6c3a4ab9819f125bcf4fbffd in term 1.
I20260928 06:28:53.844762 31311 leader_election.cc:302] T ea222aaa3635440d946303b97bbf4f83 P 6569a7bc6c3a4ab9819f125bcf4fbffd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 555fe426f3e5471ba94d6e86c80bda01, 6569a7bc6c3a4ab9819f125bcf4fbffd; no voters:
I20260928 06:28:53.844874 31461 raft_consensus.cc:2781] T ea222aaa3635440d946303b97bbf4f83 P 6569a7bc6c3a4ab9819f125bcf4fbffd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:28:53.844982 31461 raft_consensus.cc:686] T ea222aaa3635440d946303b97bbf4f83 P 6569a7bc6c3a4ab9819f125bcf4fbffd [term 1 LEADER]: Becoming Leader. State: Replica: 6569a7bc6c3a4ab9819f125bcf4fbffd, State: Running, Role: LEADER
I20260928 06:28:53.845067 31461 consensus_queue.cc:237] T ea222aaa3635440d946303b97bbf4f83 P 6569a7bc6c3a4ab9819f125bcf4fbffd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555fe426f3e5471ba94d6e86c80bda01" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 39737 } } peers { permanent_uuid: "1b186b7e369049d9a87566312419b111" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41181 } } peers { permanent_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 44361 } }
I20260928 06:28:53.845834 30941 catalog_manager.cc:5495] T ea222aaa3635440d946303b97bbf4f83 P 6569a7bc6c3a4ab9819f125bcf4fbffd reported cstate change: term changed from 0 to 1, leader changed from <none> to 6569a7bc6c3a4ab9819f125bcf4fbffd (127.22.225.131). New cstate: current_term: 1 leader_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555fe426f3e5471ba94d6e86c80bda01" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 39737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b186b7e369049d9a87566312419b111" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 44361 } health_report { overall_health: HEALTHY } } }
I20260928 06:28:53.855698 31531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:53.853256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35680 (local address 127.22.225.131:44361)
0928 06:28:53.853390 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:53.853392 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:53.853398 (+ 6us) server_negotiation.cc:407] Connection header received
0928 06:28:53.853425 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:53.853426 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:53.853441 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:53.853534 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:28:53.853670 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:53.854151 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:53.854900 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:53.855103 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:53.855178 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:53.855354 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:28:53.855450 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:28:53.855543 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:53.855581 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 06:28:53.855601 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":94,"threads_started":1}
I20260928 06:28:53.856541 31099 raft_consensus.cc:1261] T ea222aaa3635440d946303b97bbf4f83 P 555fe426f3e5471ba94d6e86c80bda01 [term 1 FOLLOWER]: Refusing update from remote peer 6569a7bc6c3a4ab9819f125bcf4fbffd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:28:53.856616 31239 raft_consensus.cc:1261] T ea222aaa3635440d946303b97bbf4f83 P 1b186b7e369049d9a87566312419b111 [term 1 FOLLOWER]: Refusing update from remote peer 6569a7bc6c3a4ab9819f125bcf4fbffd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:28:53.856824 31461 consensus_queue.cc:1035] T ea222aaa3635440d946303b97bbf4f83 P 6569a7bc6c3a4ab9819f125bcf4fbffd [LEADER]: Connected to new peer: Peer: permanent_uuid: "555fe426f3e5471ba94d6e86c80bda01" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 39737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:28:53.857017 31528 consensus_queue.cc:1035] T ea222aaa3635440d946303b97bbf4f83 P 6569a7bc6c3a4ab9819f125bcf4fbffd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b186b7e369049d9a87566312419b111" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:28:53.860612 30942 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:28:53.860699 30942 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:28:53.861249 30942 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 58908c52510e473e923860f490430124: Sending DeleteTablet for 3 replicas of tablet ea222aaa3635440d946303b97bbf4f83
I20260928 06:28:53.861563 31079 tablet_service.cc:1514] Processing DeleteTablet for tablet ea222aaa3635440d946303b97bbf4f83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:28:53 UTC) from {username='oryx', principal='oryx/127.22.225.190@KRBTEST.COM'} at 127.0.0.1:55466
I20260928 06:28:53.861568 31219 tablet_service.cc:1514] Processing DeleteTablet for tablet ea222aaa3635440d946303b97bbf4f83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:28:53 UTC) from {username='oryx', principal='oryx/127.22.225.190@KRBTEST.COM'} at 127.0.0.1:57146
I20260928 06:28:53.861649 31478 tablet_replica.cc:330] T ea222aaa3635440d946303b97bbf4f83 P 1b186b7e369049d9a87566312419b111: stopping tablet replica
I20260928 06:28:53.861660 31477 tablet_replica.cc:330] T ea222aaa3635440d946303b97bbf4f83 P 555fe426f3e5471ba94d6e86c80bda01: stopping tablet replica
I20260928 06:28:53.861698 31478 raft_consensus.cc:2229] T ea222aaa3635440d946303b97bbf4f83 P 1b186b7e369049d9a87566312419b111 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:28:53.861728 31477 raft_consensus.cc:2229] T ea222aaa3635440d946303b97bbf4f83 P 555fe426f3e5471ba94d6e86c80bda01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:28:53.861743 31478 raft_consensus.cc:2258] T ea222aaa3635440d946303b97bbf4f83 P 1b186b7e369049d9a87566312419b111 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:28:53.861779 31477 raft_consensus.cc:2258] T ea222aaa3635440d946303b97bbf4f83 P 555fe426f3e5471ba94d6e86c80bda01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:28:53.862013 31359 tablet_service.cc:1514] Processing DeleteTablet for tablet ea222aaa3635440d946303b97bbf4f83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:28:53 UTC) from {username='oryx', principal='oryx/127.22.225.190@KRBTEST.COM'} at 127.0.0.1:35664
I20260928 06:28:53.862051 31477 ts_tablet_manager.cc:1905] T ea222aaa3635440d946303b97bbf4f83 P 555fe426f3e5471ba94d6e86c80bda01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:28:53.862116 31476 tablet_replica.cc:330] T ea222aaa3635440d946303b97bbf4f83 P 6569a7bc6c3a4ab9819f125bcf4fbffd: stopping tablet replica
I20260928 06:28:53.862167 31476 raft_consensus.cc:2229] T ea222aaa3635440d946303b97bbf4f83 P 6569a7bc6c3a4ab9819f125bcf4fbffd [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:28:53.862246 31476 raft_consensus.cc:2258] T ea222aaa3635440d946303b97bbf4f83 P 6569a7bc6c3a4ab9819f125bcf4fbffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:28:53.862437 30942 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53042:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 06:28:53.862558 31476 ts_tablet_manager.cc:1905] T ea222aaa3635440d946303b97bbf4f83 P 6569a7bc6c3a4ab9819f125bcf4fbffd: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 06:28:53.862576 30942 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:28:53.862861 31478 ts_tablet_manager.cc:1905] T ea222aaa3635440d946303b97bbf4f83 P 1b186b7e369049d9a87566312419b111: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:28:53.863490 31477 ts_tablet_manager.cc:1918] T ea222aaa3635440d946303b97bbf4f83 P 555fe426f3e5471ba94d6e86c80bda01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:28:53.863541 31477 log.cc:1196] T ea222aaa3635440d946303b97bbf4f83 P 555fe426f3e5471ba94d6e86c80bda01: Deleting WAL directory at /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/ts-0/wal/wals/ea222aaa3635440d946303b97bbf4f83
I20260928 06:28:53.863770 31477 ts_tablet_manager.cc:1939] T ea222aaa3635440d946303b97bbf4f83 P 555fe426f3e5471ba94d6e86c80bda01: Deleting consensus metadata
I20260928 06:28:53.864001 31476 ts_tablet_manager.cc:1918] T ea222aaa3635440d946303b97bbf4f83 P 6569a7bc6c3a4ab9819f125bcf4fbffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:28:53.864044 31476 log.cc:1196] T ea222aaa3635440d946303b97bbf4f83 P 6569a7bc6c3a4ab9819f125bcf4fbffd: Deleting WAL directory at /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/ts-2/wal/wals/ea222aaa3635440d946303b97bbf4f83
I20260928 06:28:53.864243 31476 ts_tablet_manager.cc:1939] T ea222aaa3635440d946303b97bbf4f83 P 6569a7bc6c3a4ab9819f125bcf4fbffd: Deleting consensus metadata
I20260928 06:28:53.864449 31478 ts_tablet_manager.cc:1918] T ea222aaa3635440d946303b97bbf4f83 P 1b186b7e369049d9a87566312419b111: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:28:53.864500 31478 log.cc:1196] T ea222aaa3635440d946303b97bbf4f83 P 1b186b7e369049d9a87566312419b111: Deleting WAL directory at /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/ts-1/wal/wals/ea222aaa3635440d946303b97bbf4f83
I20260928 06:28:53.864744 31478 ts_tablet_manager.cc:1939] T ea222aaa3635440d946303b97bbf4f83 P 1b186b7e369049d9a87566312419b111: Deleting consensus metadata
I20260928 06:28:53.864992 30929 catalog_manager.cc:4841] TS 6569a7bc6c3a4ab9819f125bcf4fbffd (127.22.225.131:44361): tablet ea222aaa3635440d946303b97bbf4f83 (table test-table [id=5db8b12f8c3543b3b8e3e6fbfb0abbd5]) successfully deleted
I20260928 06:28:53.865053 30927 catalog_manager.cc:4841] TS 1b186b7e369049d9a87566312419b111 (127.22.225.130:41181): tablet ea222aaa3635440d946303b97bbf4f83 (table test-table [id=5db8b12f8c3543b3b8e3e6fbfb0abbd5]) successfully deleted
I20260928 06:28:53.865135 30927 catalog_manager.cc:4841] TS 555fe426f3e5471ba94d6e86c80bda01 (127.22.225.129:39737): tablet ea222aaa3635440d946303b97bbf4f83 (table test-table [id=5db8b12f8c3543b3b8e3e6fbfb0abbd5]) successfully deleted
I20260928 06:28:53.866074 31079 tablet_service.cc:1467] Processing CreateTablet for tablet 170d19d06947494d88e8a09983f35a2c (DEFAULT_TABLE table=test-table [id=0c4c9bb33c2f4dbfbac50a0cf6f3f83b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:28:53.866074 31359 tablet_service.cc:1467] Processing CreateTablet for tablet 170d19d06947494d88e8a09983f35a2c (DEFAULT_TABLE table=test-table [id=0c4c9bb33c2f4dbfbac50a0cf6f3f83b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:28:53.866176 31359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 170d19d06947494d88e8a09983f35a2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:53.866178 31079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 170d19d06947494d88e8a09983f35a2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:53.867260 31455 tablet_bootstrap.cc:492] T 170d19d06947494d88e8a09983f35a2c P 555fe426f3e5471ba94d6e86c80bda01: Bootstrap starting.
I20260928 06:28:53.867272 31457 tablet_bootstrap.cc:492] T 170d19d06947494d88e8a09983f35a2c P 6569a7bc6c3a4ab9819f125bcf4fbffd: Bootstrap starting.
I20260928 06:28:53.867295 31219 tablet_service.cc:1467] Processing CreateTablet for tablet 170d19d06947494d88e8a09983f35a2c (DEFAULT_TABLE table=test-table [id=0c4c9bb33c2f4dbfbac50a0cf6f3f83b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:28:53.867404 31219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 170d19d06947494d88e8a09983f35a2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:53.867583 31455 tablet_bootstrap.cc:654] T 170d19d06947494d88e8a09983f35a2c P 555fe426f3e5471ba94d6e86c80bda01: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:53.867722 31457 tablet_bootstrap.cc:654] T 170d19d06947494d88e8a09983f35a2c P 6569a7bc6c3a4ab9819f125bcf4fbffd: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:53.868098 31455 tablet_bootstrap.cc:492] T 170d19d06947494d88e8a09983f35a2c P 555fe426f3e5471ba94d6e86c80bda01: No bootstrap required, opened a new log
I20260928 06:28:53.868134 31455 ts_tablet_manager.cc:1397] T 170d19d06947494d88e8a09983f35a2c P 555fe426f3e5471ba94d6e86c80bda01: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:28:53.868229 31455 raft_consensus.cc:348] T 170d19d06947494d88e8a09983f35a2c P 555fe426f3e5471ba94d6e86c80bda01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555fe426f3e5471ba94d6e86c80bda01" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 39737 } } peers { permanent_uuid: "1b186b7e369049d9a87566312419b111" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41181 } } peers { permanent_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 44361 } }
I20260928 06:28:53.868283 31455 raft_consensus.cc:374] T 170d19d06947494d88e8a09983f35a2c P 555fe426f3e5471ba94d6e86c80bda01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:53.868312 31455 raft_consensus.cc:729] T 170d19d06947494d88e8a09983f35a2c P 555fe426f3e5471ba94d6e86c80bda01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 555fe426f3e5471ba94d6e86c80bda01, State: Initialized, Role: FOLLOWER
I20260928 06:28:53.868352 31455 consensus_queue.cc:260] T 170d19d06947494d88e8a09983f35a2c P 555fe426f3e5471ba94d6e86c80bda01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555fe426f3e5471ba94d6e86c80bda01" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 39737 } } peers { permanent_uuid: "1b186b7e369049d9a87566312419b111" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41181 } } peers { permanent_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 44361 } }
I20260928 06:28:53.868428 31455 ts_tablet_manager.cc:1428] T 170d19d06947494d88e8a09983f35a2c P 555fe426f3e5471ba94d6e86c80bda01: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:53.868878 31453 tablet_bootstrap.cc:492] T 170d19d06947494d88e8a09983f35a2c P 1b186b7e369049d9a87566312419b111: Bootstrap starting.
I20260928 06:28:53.869187 31453 tablet_bootstrap.cc:654] T 170d19d06947494d88e8a09983f35a2c P 1b186b7e369049d9a87566312419b111: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:53.869724 31453 tablet_bootstrap.cc:492] T 170d19d06947494d88e8a09983f35a2c P 1b186b7e369049d9a87566312419b111: No bootstrap required, opened a new log
I20260928 06:28:53.869791 31453 ts_tablet_manager.cc:1397] T 170d19d06947494d88e8a09983f35a2c P 1b186b7e369049d9a87566312419b111: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:28:53.869897 31453 raft_consensus.cc:348] T 170d19d06947494d88e8a09983f35a2c P 1b186b7e369049d9a87566312419b111 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555fe426f3e5471ba94d6e86c80bda01" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 39737 } } peers { permanent_uuid: "1b186b7e369049d9a87566312419b111" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41181 } } peers { permanent_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 44361 } }
I20260928 06:28:53.869987 31453 raft_consensus.cc:374] T 170d19d06947494d88e8a09983f35a2c P 1b186b7e369049d9a87566312419b111 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:53.870014 31453 raft_consensus.cc:729] T 170d19d06947494d88e8a09983f35a2c P 1b186b7e369049d9a87566312419b111 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b186b7e369049d9a87566312419b111, State: Initialized, Role: FOLLOWER
I20260928 06:28:53.870075 31453 consensus_queue.cc:260] T 170d19d06947494d88e8a09983f35a2c P 1b186b7e369049d9a87566312419b111 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555fe426f3e5471ba94d6e86c80bda01" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 39737 } } peers { permanent_uuid: "1b186b7e369049d9a87566312419b111" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41181 } } peers { permanent_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 44361 } }
I20260928 06:28:53.870172 31453 ts_tablet_manager.cc:1428] T 170d19d06947494d88e8a09983f35a2c P 1b186b7e369049d9a87566312419b111: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 06:28:53.870543 31457 tablet_bootstrap.cc:492] T 170d19d06947494d88e8a09983f35a2c P 6569a7bc6c3a4ab9819f125bcf4fbffd: No bootstrap required, opened a new log
I20260928 06:28:53.870608 31457 ts_tablet_manager.cc:1397] T 170d19d06947494d88e8a09983f35a2c P 6569a7bc6c3a4ab9819f125bcf4fbffd: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 06:28:53.870764 31457 raft_consensus.cc:348] T 170d19d06947494d88e8a09983f35a2c P 6569a7bc6c3a4ab9819f125bcf4fbffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555fe426f3e5471ba94d6e86c80bda01" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 39737 } } peers { permanent_uuid: "1b186b7e369049d9a87566312419b111" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41181 } } peers { permanent_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 44361 } }
I20260928 06:28:53.870837 31457 raft_consensus.cc:374] T 170d19d06947494d88e8a09983f35a2c P 6569a7bc6c3a4ab9819f125bcf4fbffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:53.870864 31457 raft_consensus.cc:729] T 170d19d06947494d88e8a09983f35a2c P 6569a7bc6c3a4ab9819f125bcf4fbffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6569a7bc6c3a4ab9819f125bcf4fbffd, State: Initialized, Role: FOLLOWER
I20260928 06:28:53.870903 31457 consensus_queue.cc:260] T 170d19d06947494d88e8a09983f35a2c P 6569a7bc6c3a4ab9819f125bcf4fbffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555fe426f3e5471ba94d6e86c80bda01" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 39737 } } peers { permanent_uuid: "1b186b7e369049d9a87566312419b111" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41181 } } peers { permanent_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 44361 } }
I20260928 06:28:53.870998 31457 ts_tablet_manager.cc:1428] T 170d19d06947494d88e8a09983f35a2c P 6569a7bc6c3a4ab9819f125bcf4fbffd: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 06:28:54.181681 31508 raft_consensus.cc:482] T 170d19d06947494d88e8a09983f35a2c P 555fe426f3e5471ba94d6e86c80bda01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:28:54.181800 31508 raft_consensus.cc:504] T 170d19d06947494d88e8a09983f35a2c P 555fe426f3e5471ba94d6e86c80bda01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555fe426f3e5471ba94d6e86c80bda01" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 39737 } } peers { permanent_uuid: "1b186b7e369049d9a87566312419b111" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41181 } } peers { permanent_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 44361 } }
I20260928 06:28:54.181990 31508 leader_election.cc:288] T 170d19d06947494d88e8a09983f35a2c P 555fe426f3e5471ba94d6e86c80bda01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b186b7e369049d9a87566312419b111 (127.22.225.130:41181), 6569a7bc6c3a4ab9819f125bcf4fbffd (127.22.225.131:44361)
I20260928 06:28:54.182255 31379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "170d19d06947494d88e8a09983f35a2c" candidate_uuid: "555fe426f3e5471ba94d6e86c80bda01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" is_pre_election: true
I20260928 06:28:54.182256 31239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "170d19d06947494d88e8a09983f35a2c" candidate_uuid: "555fe426f3e5471ba94d6e86c80bda01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b186b7e369049d9a87566312419b111" is_pre_election: true
I20260928 06:28:54.182348 31379 raft_consensus.cc:2454] T 170d19d06947494d88e8a09983f35a2c P 6569a7bc6c3a4ab9819f125bcf4fbffd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 555fe426f3e5471ba94d6e86c80bda01 in term 0.
I20260928 06:28:54.182386 31239 raft_consensus.cc:2454] T 170d19d06947494d88e8a09983f35a2c P 1b186b7e369049d9a87566312419b111 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 555fe426f3e5471ba94d6e86c80bda01 in term 0.
I20260928 06:28:54.182497 31033 leader_election.cc:302] T 170d19d06947494d88e8a09983f35a2c P 555fe426f3e5471ba94d6e86c80bda01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 555fe426f3e5471ba94d6e86c80bda01, 6569a7bc6c3a4ab9819f125bcf4fbffd; no voters:
I20260928 06:28:54.182574 31508 raft_consensus.cc:2781] T 170d19d06947494d88e8a09983f35a2c P 555fe426f3e5471ba94d6e86c80bda01 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:28:54.182615 31508 raft_consensus.cc:482] T 170d19d06947494d88e8a09983f35a2c P 555fe426f3e5471ba94d6e86c80bda01 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:28:54.182632 31508 raft_consensus.cc:3037] T 170d19d06947494d88e8a09983f35a2c P 555fe426f3e5471ba94d6e86c80bda01 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:54.183506 31508 raft_consensus.cc:504] T 170d19d06947494d88e8a09983f35a2c P 555fe426f3e5471ba94d6e86c80bda01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555fe426f3e5471ba94d6e86c80bda01" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 39737 } } peers { permanent_uuid: "1b186b7e369049d9a87566312419b111" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41181 } } peers { permanent_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 44361 } }
I20260928 06:28:54.183630 31508 leader_election.cc:288] T 170d19d06947494d88e8a09983f35a2c P 555fe426f3e5471ba94d6e86c80bda01 [CANDIDATE]: Term 1 election: Requested vote from peers 1b186b7e369049d9a87566312419b111 (127.22.225.130:41181), 6569a7bc6c3a4ab9819f125bcf4fbffd (127.22.225.131:44361)
I20260928 06:28:54.183780 31239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "170d19d06947494d88e8a09983f35a2c" candidate_uuid: "555fe426f3e5471ba94d6e86c80bda01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b186b7e369049d9a87566312419b111"
I20260928 06:28:54.183810 31379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "170d19d06947494d88e8a09983f35a2c" candidate_uuid: "555fe426f3e5471ba94d6e86c80bda01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd"
I20260928 06:28:54.183866 31379 raft_consensus.cc:3037] T 170d19d06947494d88e8a09983f35a2c P 6569a7bc6c3a4ab9819f125bcf4fbffd [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:54.183887 31239 raft_consensus.cc:3037] T 170d19d06947494d88e8a09983f35a2c P 1b186b7e369049d9a87566312419b111 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:54.184554 31239 raft_consensus.cc:2454] T 170d19d06947494d88e8a09983f35a2c P 1b186b7e369049d9a87566312419b111 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 555fe426f3e5471ba94d6e86c80bda01 in term 1.
I20260928 06:28:54.184598 31379 raft_consensus.cc:2454] T 170d19d06947494d88e8a09983f35a2c P 6569a7bc6c3a4ab9819f125bcf4fbffd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 555fe426f3e5471ba94d6e86c80bda01 in term 1.
I20260928 06:28:54.184765 31033 leader_election.cc:302] T 170d19d06947494d88e8a09983f35a2c P 555fe426f3e5471ba94d6e86c80bda01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 555fe426f3e5471ba94d6e86c80bda01, 6569a7bc6c3a4ab9819f125bcf4fbffd; no voters:
I20260928 06:28:54.184862 31508 raft_consensus.cc:2781] T 170d19d06947494d88e8a09983f35a2c P 555fe426f3e5471ba94d6e86c80bda01 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:28:54.184979 31508 raft_consensus.cc:686] T 170d19d06947494d88e8a09983f35a2c P 555fe426f3e5471ba94d6e86c80bda01 [term 1 LEADER]: Becoming Leader. State: Replica: 555fe426f3e5471ba94d6e86c80bda01, State: Running, Role: LEADER
I20260928 06:28:54.185029 31508 consensus_queue.cc:237] T 170d19d06947494d88e8a09983f35a2c P 555fe426f3e5471ba94d6e86c80bda01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555fe426f3e5471ba94d6e86c80bda01" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 39737 } } peers { permanent_uuid: "1b186b7e369049d9a87566312419b111" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41181 } } peers { permanent_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 44361 } }
I20260928 06:28:54.185622 30941 catalog_manager.cc:5495] T 170d19d06947494d88e8a09983f35a2c P 555fe426f3e5471ba94d6e86c80bda01 reported cstate change: term changed from 0 to 1, leader changed from <none> to 555fe426f3e5471ba94d6e86c80bda01 (127.22.225.129). New cstate: current_term: 1 leader_uuid: "555fe426f3e5471ba94d6e86c80bda01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555fe426f3e5471ba94d6e86c80bda01" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 39737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b186b7e369049d9a87566312419b111" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 44361 } health_report { overall_health: UNKNOWN } } }
I20260928 06:28:54.225488 31450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:54.223253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55478 (local address 127.22.225.129:39737)
0928 06:28:54.223298 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:54.223300 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:54.223308 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:28:54.223351 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:54.223352 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:54.223363 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:54.223412 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:28:54.223513 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:54.223984 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:54.224794 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:54.225001 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:54.225084 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:54.225232 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:28:54.225266 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:28:54.225315 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:54.225381 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 06:28:54.225405 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 06:28:54.227763 31239 raft_consensus.cc:1261] T 170d19d06947494d88e8a09983f35a2c P 1b186b7e369049d9a87566312419b111 [term 1 FOLLOWER]: Refusing update from remote peer 555fe426f3e5471ba94d6e86c80bda01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:28:54.227787 31379 raft_consensus.cc:1261] T 170d19d06947494d88e8a09983f35a2c P 6569a7bc6c3a4ab9819f125bcf4fbffd [term 1 FOLLOWER]: Refusing update from remote peer 555fe426f3e5471ba94d6e86c80bda01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:28:54.228034 31508 consensus_queue.cc:1035] T 170d19d06947494d88e8a09983f35a2c P 555fe426f3e5471ba94d6e86c80bda01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6569a7bc6c3a4ab9819f125bcf4fbffd" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 44361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:28:54.228166 31539 consensus_queue.cc:1035] T 170d19d06947494d88e8a09983f35a2c P 555fe426f3e5471ba94d6e86c80bda01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b186b7e369049d9a87566312419b111" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:28:54.241358 30941 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:28:54.241479 30941 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:28:54.242107 30941 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 58908c52510e473e923860f490430124: Sending DeleteTablet for 3 replicas of tablet 170d19d06947494d88e8a09983f35a2c
I20260928 06:28:54.242344 31079 tablet_service.cc:1514] Processing DeleteTablet for tablet 170d19d06947494d88e8a09983f35a2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:28:54 UTC) from {username='oryx', principal='oryx/127.22.225.190@KRBTEST.COM'} at 127.0.0.1:55466
I20260928 06:28:54.242374 31219 tablet_service.cc:1514] Processing DeleteTablet for tablet 170d19d06947494d88e8a09983f35a2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:28:54 UTC) from {username='oryx', principal='oryx/127.22.225.190@KRBTEST.COM'} at 127.0.0.1:57146
I20260928 06:28:54.242385 31359 tablet_service.cc:1514] Processing DeleteTablet for tablet 170d19d06947494d88e8a09983f35a2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:28:54 UTC) from {username='oryx', principal='oryx/127.22.225.190@KRBTEST.COM'} at 127.0.0.1:35664
I20260928 06:28:54.242484 31477 tablet_replica.cc:330] T 170d19d06947494d88e8a09983f35a2c P 555fe426f3e5471ba94d6e86c80bda01: stopping tablet replica
I20260928 06:28:54.242496 31478 tablet_replica.cc:330] T 170d19d06947494d88e8a09983f35a2c P 1b186b7e369049d9a87566312419b111: stopping tablet replica
I20260928 06:28:54.242547 31478 raft_consensus.cc:2229] T 170d19d06947494d88e8a09983f35a2c P 1b186b7e369049d9a87566312419b111 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:28:54.242547 31477 raft_consensus.cc:2229] T 170d19d06947494d88e8a09983f35a2c P 555fe426f3e5471ba94d6e86c80bda01 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:28:54.242583 31478 raft_consensus.cc:2258] T 170d19d06947494d88e8a09983f35a2c P 1b186b7e369049d9a87566312419b111 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:28:54.242614 31477 raft_consensus.cc:2258] T 170d19d06947494d88e8a09983f35a2c P 555fe426f3e5471ba94d6e86c80bda01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:28:54.242643 31476 tablet_replica.cc:330] T 170d19d06947494d88e8a09983f35a2c P 6569a7bc6c3a4ab9819f125bcf4fbffd: stopping tablet replica
I20260928 06:28:54.242692 31476 raft_consensus.cc:2229] T 170d19d06947494d88e8a09983f35a2c P 6569a7bc6c3a4ab9819f125bcf4fbffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:28:54.242728 31476 raft_consensus.cc:2258] T 170d19d06947494d88e8a09983f35a2c P 6569a7bc6c3a4ab9819f125bcf4fbffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:28:54.242942 31477 ts_tablet_manager.cc:1905] T 170d19d06947494d88e8a09983f35a2c P 555fe426f3e5471ba94d6e86c80bda01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:28:54.243052 31478 ts_tablet_manager.cc:1905] T 170d19d06947494d88e8a09983f35a2c P 1b186b7e369049d9a87566312419b111: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:28:54.244701 31478 ts_tablet_manager.cc:1918] T 170d19d06947494d88e8a09983f35a2c P 1b186b7e369049d9a87566312419b111: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:28:54.244706 31476 ts_tablet_manager.cc:1905] T 170d19d06947494d88e8a09983f35a2c P 6569a7bc6c3a4ab9819f125bcf4fbffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:28:54.244764 31478 log.cc:1196] T 170d19d06947494d88e8a09983f35a2c P 1b186b7e369049d9a87566312419b111: Deleting WAL directory at /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/ts-1/wal/wals/170d19d06947494d88e8a09983f35a2c
I20260928 06:28:54.244975 31478 ts_tablet_manager.cc:1939] T 170d19d06947494d88e8a09983f35a2c P 1b186b7e369049d9a87566312419b111: Deleting consensus metadata
I20260928 06:28:54.245287 30927 catalog_manager.cc:4841] TS 1b186b7e369049d9a87566312419b111 (127.22.225.130:41181): tablet 170d19d06947494d88e8a09983f35a2c (table test-table [id=0c4c9bb33c2f4dbfbac50a0cf6f3f83b]) successfully deleted
I20260928 06:28:54.246112 31476 ts_tablet_manager.cc:1918] T 170d19d06947494d88e8a09983f35a2c P 6569a7bc6c3a4ab9819f125bcf4fbffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:28:54.246158 31476 log.cc:1196] T 170d19d06947494d88e8a09983f35a2c P 6569a7bc6c3a4ab9819f125bcf4fbffd: Deleting WAL directory at /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576899143097-23430-0/minicluster-data/ts-2/wal/wals/170d19d06947494d88e8a09983f35a2c
I20260928 06:28:54.246383 31476 ts_tablet_manager.cc:1939] T 170d19d06947494d88e8a09983f35a2c P 6569a7bc6c3a4ab9819f125bcf4fbffd: Deleting consensus metadata
I20260928 06:28:54.246809 30929 catalog_manager.cc:4841] TS 6569a7bc6c3a4ab9819f125bcf4fbffd (127.22.225.131:44361): tablet 170d19d06947494d88e8a09983f35a2c (table test-table [id=0c4c9bb33c2f4dbfbac50a0cf6f3f83b]) successfully deleted
I20260928 06:28:54.246922 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 31014
W20260928 06:28:54.253877 30927 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.225.129:39737 (error 108)
I20260928 06:28:54.254456 31549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:54.254260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.129:39737 (local address 127.0.0.1:55490)
0928 06:28:54.254363 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:54.254414 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.129:39737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":45,"threads_started":1}
W20260928 06:28:54.254594 30927 catalog_manager.cc:4584] TS 555fe426f3e5471ba94d6e86c80bda01 (127.22.225.129:39737): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 170d19d06947494d88e8a09983f35a2c: Network error: Client connection negotiation failed: client connection to 127.22.225.129:39737: connect: Connection refused (error 111)
I20260928 06:28:54.254642 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 31154
I20260928 06:28:54.262296 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 31294
I20260928 06:28:54.268512 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 30910
2026-09-28T06:28:54Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3199 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790576899143097-23430-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:28:54 dist-test-slave-d59m krb5kdc[31552](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:28:54 dist-test-slave-d59m krb5kdc[31552](info): set up 2 sockets
Sep 28 06:28:54 dist-test-slave-d59m krb5kdc[31552](info): commencing operation
krb5kdc: starting...
W20260928 06:28:55.325876 23430 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Sep 28 06:28:55 dist-test-slave-d59m krb5kdc[31552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576935, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:28:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:28:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab.
Entry for principal kudu/127.22.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab.
I20260928 06:28:55.451618 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790576899143097-23430-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790576899143097-23430-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790576899143097-23430-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:41387
--webserver_interface=127.22.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab
--principal=kudu/127.22.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:33629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.225.190:41387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:28:55.551008 31565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:28:55.551211 31565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:28:55.551234 31565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:28:55.552371 31565 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:28:55.552412 31565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:28:55.552425 31565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:28:55.552438 31565 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:28:55.552489 31565 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:28:55.553990 31565 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:33629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790576899143097-23430-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790576899143097-23430-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.225.190:41387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:41387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31565
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790576899143097-23430-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:28:55.554239 31565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:28:55.554481 31565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:28:55.559376 31570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:55.559526 31571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:55.559448 31565 server_base.cc:1034] running on GCE node
W20260928 06:28:55.559399 31573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:55.559914 31565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:28:55.560252 31565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:28:55.561426 31565 hybrid_clock.cc:648] HybridClock initialized: now 1790576935561412 us; error 29 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T06:28:55Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1300 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:28:55 dist-test-slave-d59m krb5kdc[31578](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:28:55 dist-test-slave-d59m krb5kdc[31578](info): set up 2 sockets
Sep 28 06:28:55 dist-test-slave-d59m krb5kdc[31578](info): commencing operation
krb5kdc: starting...
W20260928 06:28:56.654136 23430 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Sep 28 06:28:56 dist-test-slave-d59m krb5kdc[31578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:28:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:28:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab.
Entry for principal kudu/127.22.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab.
I20260928 06:28:56.783116 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:40039
--webserver_interface=127.22.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab
--principal=kudu/127.22.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:36285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.225.190:40039
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:28:56.863792 31591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:28:56.864001 31591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:28:56.864028 31591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:28:56.865228 31591 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:28:56.865269 31591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:28:56.865283 31591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:28:56.865296 31591 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:28:56.865340 31591 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:28:56.866959 31591 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:36285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.225.190:40039
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab
--principal=kudu/127.22.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:40039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31591
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:28:56.867202 31591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:28:56.867427 31591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:28:56.872224 31597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:56.872402 31591 server_base.cc:1034] running on GCE node
W20260928 06:28:56.872439 31596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:56.872251 31599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:56.872777 31591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:28:56.873095 31591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:28:56.874257 31591 hybrid_clock.cc:648] HybridClock initialized: now 1790576936874246 us; error 28 us; skew 500 ppm
Sep 28 06:28:56 dist-test-slave-d59m krb5kdc[31578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576936, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:28:56.877140 31591 init.cc:377] Logged in from keytab as kudu/127.22.225.190@KRBTEST.COM (short username kudu)
I20260928 06:28:56.877753 31591 webserver.cc:460] Webserver started at http://127.22.225.190:40741/ using document root <none> and password file <none>
I20260928 06:28:56.878077 31591 fs_manager.cc:359] Metadata directory not provided
I20260928 06:28:56.878152 31591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:28:56.878309 31591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:28:56.879869 31591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/master-0/data/instance:
uuid: "c94b06e2757140948bc74d9b1dd41cf1"
format_stamp: "Formatted at 2026-09-28 06:28:56 on dist-test-slave-d59m"
server_key: "4691bb3888f496c359e6b19776ab85a9"
server_key_iv: "345c8a2c06e26cf9c76f1876d0d70d6f"
server_key_version: "encryptionkey@0"
I20260928 06:28:56.880357 31591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/master-0/wal/instance:
uuid: "c94b06e2757140948bc74d9b1dd41cf1"
format_stamp: "Formatted at 2026-09-28 06:28:56 on dist-test-slave-d59m"
server_key: "4691bb3888f496c359e6b19776ab85a9"
server_key_iv: "345c8a2c06e26cf9c76f1876d0d70d6f"
server_key_version: "encryptionkey@0"
I20260928 06:28:56.882357 31591 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260928 06:28:56.883347 31606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:56.883574 31591 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:28:56.883697 31591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/master-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/master-0/wal
uuid: "c94b06e2757140948bc74d9b1dd41cf1"
format_stamp: "Formatted at 2026-09-28 06:28:56 on dist-test-slave-d59m"
server_key: "4691bb3888f496c359e6b19776ab85a9"
server_key_iv: "345c8a2c06e26cf9c76f1876d0d70d6f"
server_key_version: "encryptionkey@0"
I20260928 06:28:56.883781 31591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:28:56.900349 31591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:28:56.903120 31591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:28:56.903299 31591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:28:56.908399 31591 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.190:40039
I20260928 06:28:56.908424 31658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.190:40039 every 8 connection(s)
I20260928 06:28:56.908852 31591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
I20260928 06:28:56.909740 31659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:56.912005 31659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c94b06e2757140948bc74d9b1dd41cf1: Bootstrap starting.
I20260928 06:28:56.912678 31659 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c94b06e2757140948bc74d9b1dd41cf1: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:56.912962 31659 log.cc:824] T 00000000000000000000000000000000 P c94b06e2757140948bc74d9b1dd41cf1: Log is configured to *not* fsync() on all Append() calls
I20260928 06:28:56.913740 31659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c94b06e2757140948bc74d9b1dd41cf1: No bootstrap required, opened a new log
I20260928 06:28:56.915298 31659 raft_consensus.cc:348] T 00000000000000000000000000000000 P c94b06e2757140948bc74d9b1dd41cf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c94b06e2757140948bc74d9b1dd41cf1" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 40039 } }
I20260928 06:28:56.915464 31659 raft_consensus.cc:374] T 00000000000000000000000000000000 P c94b06e2757140948bc74d9b1dd41cf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:28:56.915529 31659 raft_consensus.cc:729] T 00000000000000000000000000000000 P c94b06e2757140948bc74d9b1dd41cf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c94b06e2757140948bc74d9b1dd41cf1, State: Initialized, Role: FOLLOWER
I20260928 06:28:56.915709 31659 consensus_queue.cc:260] T 00000000000000000000000000000000 P c94b06e2757140948bc74d9b1dd41cf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c94b06e2757140948bc74d9b1dd41cf1" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 40039 } }
I20260928 06:28:56.915778 31659 raft_consensus.cc:388] T 00000000000000000000000000000000 P c94b06e2757140948bc74d9b1dd41cf1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:28:56.915831 31659 raft_consensus.cc:482] T 00000000000000000000000000000000 P c94b06e2757140948bc74d9b1dd41cf1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:28:56.915880 31659 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c94b06e2757140948bc74d9b1dd41cf1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:56.916433 31659 raft_consensus.cc:504] T 00000000000000000000000000000000 P c94b06e2757140948bc74d9b1dd41cf1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c94b06e2757140948bc74d9b1dd41cf1" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 40039 } }
I20260928 06:28:56.916522 31659 leader_election.cc:302] T 00000000000000000000000000000000 P c94b06e2757140948bc74d9b1dd41cf1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c94b06e2757140948bc74d9b1dd41cf1; no voters:
I20260928 06:28:56.916734 31659 leader_election.cc:288] T 00000000000000000000000000000000 P c94b06e2757140948bc74d9b1dd41cf1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:28:56.916791 31662 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c94b06e2757140948bc74d9b1dd41cf1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:28:56.916960 31662 raft_consensus.cc:686] T 00000000000000000000000000000000 P c94b06e2757140948bc74d9b1dd41cf1 [term 1 LEADER]: Becoming Leader. State: Replica: c94b06e2757140948bc74d9b1dd41cf1, State: Running, Role: LEADER
I20260928 06:28:56.916997 31659 sys_catalog.cc:564] T 00000000000000000000000000000000 P c94b06e2757140948bc74d9b1dd41cf1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:28:56.917116 31662 consensus_queue.cc:237] T 00000000000000000000000000000000 P c94b06e2757140948bc74d9b1dd41cf1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c94b06e2757140948bc74d9b1dd41cf1" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 40039 } }
I20260928 06:28:56.917524 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 31591
I20260928 06:28:56.917594 31664 sys_catalog.cc:455] T 00000000000000000000000000000000 P c94b06e2757140948bc74d9b1dd41cf1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c94b06e2757140948bc74d9b1dd41cf1. Latest consensus state: current_term: 1 leader_uuid: "c94b06e2757140948bc74d9b1dd41cf1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c94b06e2757140948bc74d9b1dd41cf1" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 40039 } } }
I20260928 06:28:56.917658 31664 sys_catalog.cc:458] T 00000000000000000000000000000000 P c94b06e2757140948bc74d9b1dd41cf1 [sys.catalog]: This master's current role is: LEADER
I20260928 06:28:56.917657 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/master-0/wal/instance
I20260928 06:28:56.917805 31663 sys_catalog.cc:455] T 00000000000000000000000000000000 P c94b06e2757140948bc74d9b1dd41cf1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c94b06e2757140948bc74d9b1dd41cf1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c94b06e2757140948bc74d9b1dd41cf1" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 40039 } } }
I20260928 06:28:56.917843 31663 sys_catalog.cc:458] T 00000000000000000000000000000000 P c94b06e2757140948bc74d9b1dd41cf1 [sys.catalog]: This master's current role is: LEADER
I20260928 06:28:56.917868 23430 external_mini_cluster.cc:1428] Setting key 6cbb9112a2debce973cc9bbd5c81af83
I20260928 06:28:56.918303 31670 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:28:56.918943 31670 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:28:56.922847 31670 catalog_manager.cc:1347] Generated new cluster ID: 14e64f749762418ea449f4046d5278d5
I20260928 06:28:56.922899 31670 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:28:56 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:28:56.929585 31679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:56.918557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33500 (local address 127.22.225.190:40039)
0928 06:28:56.918710 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:56.918719 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:56.918970 (+ 251us) server_negotiation.cc:407] Connection header received
0928 06:28:56.919220 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:56.919229 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:56.919290 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:56.919432 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:56.920372 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:56.921348 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:56.922127 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:56.922741 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:56.922842 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:56.925683 (+ 2841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:56.925687 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:56.925688 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:56.925700 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:56.927667 (+ 1967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:56.928019 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:56.928022 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:56.928022 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:56.928132 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:56.928298 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:56.928298 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:56.928299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:56.928628 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:56.928766 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:56.929223 (+ 457us) server_negotiation.cc:299] Negotiation successful
0928 06:28:56.929354 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":60,"threads_started":1}
I20260928 06:28:56.945294 31670 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:28:56.945894 31670 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:28:56.949702 31670 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c94b06e2757140948bc74d9b1dd41cf1: Generated new TSK 0
I20260928 06:28:56.949889 31670 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.225.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab.
Entry for principal kudu/127.22.225.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab.
I20260928 06:28:56.986891 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.129:0
--local_ip_for_outbound_sockets=127.22.225.129
--webserver_interface=127.22.225.129
--webserver_port=0
--tserver_master_addrs=127.22.225.190:40039
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab
--principal=kudu/127.22.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:36285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:28:57.106172 31685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:28:57.106470 31685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:28:57.106504 31685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:28:57.108162 31685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:28:57.108224 31685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:28:57.108263 31685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.129
I20260928 06:28:57.110275 31685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:36285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab
--principal=kudu/127.22.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.225.129
--webserver_port=0
--tserver_master_addrs=127.22.225.190:40039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.129
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:28:57.110533 31685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:28:57.110790 31685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:28:57.116008 31693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:57.116065 31691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:57.116012 31690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:57.116379 31685 server_base.cc:1034] running on GCE node
I20260928 06:28:57.116638 31685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:28:57.116976 31685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:28:57.118176 31685 hybrid_clock.cc:648] HybridClock initialized: now 1790576937118162 us; error 30 us; skew 500 ppm
Sep 28 06:28:57 dist-test-slave-d59m krb5kdc[31578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576937, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:28:57.120828 31685 init.cc:377] Logged in from keytab as kudu/127.22.225.129@KRBTEST.COM (short username kudu)
I20260928 06:28:57.121275 31685 webserver.cc:460] Webserver started at http://127.22.225.129:46341/ using document root <none> and password file <none>
I20260928 06:28:57.121508 31685 fs_manager.cc:359] Metadata directory not provided
I20260928 06:28:57.121553 31685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:28:57.121681 31685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:28:57.122777 31685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-0/data/instance:
uuid: "9a5de9d8fe35416ea912d2ab0da02a45"
format_stamp: "Formatted at 2026-09-28 06:28:57 on dist-test-slave-d59m"
server_key: "25d7ab184bc26929490e3862dbc4896d"
server_key_iv: "0ff531a35cc5fb5c00fa36a229e146ee"
server_key_version: "encryptionkey@0"
I20260928 06:28:57.123164 31685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-0/wal/instance:
uuid: "9a5de9d8fe35416ea912d2ab0da02a45"
format_stamp: "Formatted at 2026-09-28 06:28:57 on dist-test-slave-d59m"
server_key: "25d7ab184bc26929490e3862dbc4896d"
server_key_iv: "0ff531a35cc5fb5c00fa36a229e146ee"
server_key_version: "encryptionkey@0"
I20260928 06:28:57.124518 31685 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:28:57.125324 31700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:57.125510 31685 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:28:57.125602 31685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-0/wal
uuid: "9a5de9d8fe35416ea912d2ab0da02a45"
format_stamp: "Formatted at 2026-09-28 06:28:57 on dist-test-slave-d59m"
server_key: "25d7ab184bc26929490e3862dbc4896d"
server_key_iv: "0ff531a35cc5fb5c00fa36a229e146ee"
server_key_version: "encryptionkey@0"
I20260928 06:28:57.125690 31685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:28:57.151413 31685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:28:57.154186 31685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:28:57.154364 31685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:28:57.154614 31685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:28:57.154950 31685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:28:57.155006 31685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:57.155050 31685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:28:57.155076 31685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:57.161650 31685 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.129:33011
I20260928 06:28:57.161702 31813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.129:33011 every 8 connection(s)
I20260928 06:28:57.163029 31685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
I20260928 06:28:57.163460 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 31685
I20260928 06:28:57.163573 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-0/wal/instance
I20260928 06:28:57.163748 23430 external_mini_cluster.cc:1428] Setting key 0ffd813261e8430363241248f1eea347
Sep 28 06:28:57 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576937, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.129@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.225.130@KRBTEST.COM; defaulting to no policy
I20260928 06:28:57.175410 31679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:57.163923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:58385 (local address 127.22.225.190:40039)
0928 06:28:57.164086 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:57.164089 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:57.164189 (+ 100us) server_negotiation.cc:407] Connection header received
0928 06:28:57.164982 (+ 793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:57.164984 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:57.164999 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:57.165073 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:57.166523 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:57.167116 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:57.168664 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:57.168891 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:57.168953 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:57.171958 (+ 3005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:57.171959 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:57.171959 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:57.171968 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:57.173775 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:57.174318 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:57.174320 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:57.174320 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:57.174386 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:57.174679 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:57.174679 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:57.174679 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:57.174780 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:57.174830 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:57.175193 (+ 363us) server_negotiation.cc:299] Negotiation successful
0928 06:28:57.175268 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.130@KRBTEST.COM" created.
I20260928 06:28:57.175614 31816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:57.163898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:40039 (local address 127.22.225.129:58385)
0928 06:28:57.164066 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:57.164112 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:57.164845 (+ 733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:57.165156 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:57.165162 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:57.166084 (+ 922us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:57.166397 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:57.166404 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:57.167222 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:57.167223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:57.168549 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:57.168552 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:57.168987 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:57.168988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:57.169139 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:57.169816 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:57.169827 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:57.171836 (+ 2009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:57.173880 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:57.173888 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:57.173890 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:57.174233 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:57.174461 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:57.174462 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:57.174462 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:57.174611 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:57.174844 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:57.174850 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:57.175047 (+ 197us) client_negotiation.cc:769] Sending connection context
0928 06:28:57.175172 (+ 125us) client_negotiation.cc:240] Negotiation successful
0928 06:28:57.175353 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":85,"threads_started":1}
I20260928 06:28:57.176121 31814 heartbeater.cc:347] Connected to a master server at 127.22.225.190:40039
I20260928 06:28:57.176239 31814 heartbeater.cc:464] Registering TS with master...
I20260928 06:28:57.176433 31814 heartbeater.cc:511] Master 127.22.225.190:40039 requested a full tablet report, sending...
I20260928 06:28:57.177011 31623 ts_manager.cc:194] Registered new tserver with Master: 9a5de9d8fe35416ea912d2ab0da02a45 (127.22.225.129:33011)
I20260928 06:28:57.177631 31623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.225.129@KRBTEST.COM'} at 127.22.225.129:58385
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab.
Entry for principal kudu/127.22.225.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab.
I20260928 06:28:57.191004 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.130:0
--local_ip_for_outbound_sockets=127.22.225.130
--webserver_interface=127.22.225.130
--webserver_port=0
--tserver_master_addrs=127.22.225.190:40039
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab
--principal=kudu/127.22.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:36285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:28:57.311851 31819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:28:57.312136 31819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:28:57.312177 31819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:28:57.313918 31819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:28:57.313980 31819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:28:57.314021 31819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.130
I20260928 06:28:57.316191 31819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:36285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab
--principal=kudu/127.22.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.225.130
--webserver_port=0
--tserver_master_addrs=127.22.225.190:40039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.130
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:28:57.316463 31819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:28:57.316771 31819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:28:57.322186 31827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:57.322207 31825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:57.322194 31824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:57.322511 31819 server_base.cc:1034] running on GCE node
I20260928 06:28:57.322727 31819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:28:57.323021 31819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:28:57.324209 31819 hybrid_clock.cc:648] HybridClock initialized: now 1790576937324193 us; error 44 us; skew 500 ppm
Sep 28 06:28:57 dist-test-slave-d59m krb5kdc[31578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576937, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:28:57.327252 31819 init.cc:377] Logged in from keytab as kudu/127.22.225.130@KRBTEST.COM (short username kudu)
I20260928 06:28:57.327704 31819 webserver.cc:460] Webserver started at http://127.22.225.130:32957/ using document root <none> and password file <none>
I20260928 06:28:57.327951 31819 fs_manager.cc:359] Metadata directory not provided
I20260928 06:28:57.328022 31819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:28:57.328168 31819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:28:57.329432 31819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-1/data/instance:
uuid: "2186c977b9be4d8a9c16d6bdf139d872"
format_stamp: "Formatted at 2026-09-28 06:28:57 on dist-test-slave-d59m"
server_key: "74bbeedeaaff13f9c1eb7855d9a98a93"
server_key_iv: "87d31fc7fd6edce503f89ddf9b96103c"
server_key_version: "encryptionkey@0"
I20260928 06:28:57.329793 31819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-1/wal/instance:
uuid: "2186c977b9be4d8a9c16d6bdf139d872"
format_stamp: "Formatted at 2026-09-28 06:28:57 on dist-test-slave-d59m"
server_key: "74bbeedeaaff13f9c1eb7855d9a98a93"
server_key_iv: "87d31fc7fd6edce503f89ddf9b96103c"
server_key_version: "encryptionkey@0"
I20260928 06:28:57.331175 31819 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:28:57.331967 31834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:57.332141 31819 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:28:57.332244 31819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-1/wal
uuid: "2186c977b9be4d8a9c16d6bdf139d872"
format_stamp: "Formatted at 2026-09-28 06:28:57 on dist-test-slave-d59m"
server_key: "74bbeedeaaff13f9c1eb7855d9a98a93"
server_key_iv: "87d31fc7fd6edce503f89ddf9b96103c"
server_key_version: "encryptionkey@0"
I20260928 06:28:57.332319 31819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:28:57.344538 31819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:28:57.347441 31819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:28:57.347628 31819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:28:57.347895 31819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:28:57.348268 31819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:28:57.348315 31819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:57.348348 31819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:28:57.348377 31819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:57.354606 31819 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.130:33687
I20260928 06:28:57.354648 31947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.130:33687 every 8 connection(s)
I20260928 06:28:57.355005 31819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
I20260928 06:28:57.356746 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 31819
I20260928 06:28:57.356853 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-1/wal/instance
I20260928 06:28:57.357059 23430 external_mini_cluster.cc:1428] Setting key 5e91c4f480d539d3ebc1527ff383a0b9
Sep 28 06:28:57 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576937, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.130@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:28:57.365798 31679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:57.355743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:57407 (local address 127.22.225.190:40039)
0928 06:28:57.355788 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:57.355790 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:57.356027 (+ 237us) server_negotiation.cc:407] Connection header received
0928 06:28:57.356555 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:57.356556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:57.356569 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:57.356652 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:57.357900 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:57.358393 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:57.359725 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:57.360073 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:57.360156 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:57.362398 (+ 2242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:57.362399 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:57.362400 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:57.362407 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:57.364098 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:57.364657 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:57.364658 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:57.364659 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:57.364722 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:57.365024 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:57.365025 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:57.365025 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:57.365161 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:57.365232 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:57.365596 (+ 364us) server_negotiation.cc:299] Negotiation successful
0928 06:28:57.365686 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:28:57.366062 31950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:57.355723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:40039 (local address 127.22.225.130:57407)
0928 06:28:57.355900 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:57.355936 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:57.356446 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:57.356737 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:57.356741 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:57.357539 (+ 798us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:57.357811 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:57.357818 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:57.358503 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:57.358504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:57.359620 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:57.359623 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:57.360175 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:57.360176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:57.360283 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:57.360908 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:57.360916 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:57.362303 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:57.364194 (+ 1891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:57.364202 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:57.364204 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:57.364537 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:57.364802 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:57.364803 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:57.364803 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:57.364944 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:57.365240 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:57.365248 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:57.365422 (+ 174us) client_negotiation.cc:769] Sending connection context
0928 06:28:57.365564 (+ 142us) client_negotiation.cc:240] Negotiation successful
0928 06:28:57.365806 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":86,"threads_started":1}
WARNING: no policy specified for kudu/127.22.225.131@KRBTEST.COM; defaulting to no policy
I20260928 06:28:57.366776 31948 heartbeater.cc:347] Connected to a master server at 127.22.225.190:40039
I20260928 06:28:57.366887 31948 heartbeater.cc:464] Registering TS with master...
I20260928 06:28:57.367072 31948 heartbeater.cc:511] Master 127.22.225.190:40039 requested a full tablet report, sending...
I20260928 06:28:57.367523 31623 ts_manager.cc:194] Registered new tserver with Master: 2186c977b9be4d8a9c16d6bdf139d872 (127.22.225.130:33687)
I20260928 06:28:57.367903 31623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.225.130@KRBTEST.COM'} at 127.22.225.130:57407
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab.
Entry for principal kudu/127.22.225.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab.
I20260928 06:28:57.384217 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.131:0
--local_ip_for_outbound_sockets=127.22.225.131
--webserver_interface=127.22.225.131
--webserver_port=0
--tserver_master_addrs=127.22.225.190:40039
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab
--principal=kudu/127.22.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:36285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:28:57.469408 31953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:28:57.469617 31953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:28:57.469655 31953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:28:57.470791 31953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:28:57.470852 31953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:28:57.470894 31953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.131
I20260928 06:28:57.472313 31953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:36285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab
--principal=kudu/127.22.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.225.131
--webserver_port=0
--tserver_master_addrs=127.22.225.190:40039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.131
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:28:57.472630 31953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:28:57.472854 31953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:28:57.478302 31953 server_base.cc:1034] running on GCE node
W20260928 06:28:57.478332 31958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:57.478260 31959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:57.478590 31961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:57.478783 31953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:28:57.479019 31953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:28:57.480175 31953 hybrid_clock.cc:648] HybridClock initialized: now 1790576937480162 us; error 29 us; skew 500 ppm
Sep 28 06:28:57 dist-test-slave-d59m krb5kdc[31578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576937, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:28:57.482677 31953 init.cc:377] Logged in from keytab as kudu/127.22.225.131@KRBTEST.COM (short username kudu)
I20260928 06:28:57.483114 31953 webserver.cc:460] Webserver started at http://127.22.225.131:33137/ using document root <none> and password file <none>
I20260928 06:28:57.483335 31953 fs_manager.cc:359] Metadata directory not provided
I20260928 06:28:57.483400 31953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:28:57.483537 31953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:28:57.484714 31953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-2/data/instance:
uuid: "4ee0d29c8a414813a2bd0adbfa0a0350"
format_stamp: "Formatted at 2026-09-28 06:28:57 on dist-test-slave-d59m"
server_key: "f1609bd26aae734a74b78793826cf8ee"
server_key_iv: "2ce638761fc986a0731976a190a2989c"
server_key_version: "encryptionkey@0"
I20260928 06:28:57.485069 31953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-2/wal/instance:
uuid: "4ee0d29c8a414813a2bd0adbfa0a0350"
format_stamp: "Formatted at 2026-09-28 06:28:57 on dist-test-slave-d59m"
server_key: "f1609bd26aae734a74b78793826cf8ee"
server_key_iv: "2ce638761fc986a0731976a190a2989c"
server_key_version: "encryptionkey@0"
I20260928 06:28:57.486439 31953 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 06:28:57.487205 31968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:57.487373 31953 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:28:57.487480 31953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-2/wal
uuid: "4ee0d29c8a414813a2bd0adbfa0a0350"
format_stamp: "Formatted at 2026-09-28 06:28:57 on dist-test-slave-d59m"
server_key: "f1609bd26aae734a74b78793826cf8ee"
server_key_iv: "2ce638761fc986a0731976a190a2989c"
server_key_version: "encryptionkey@0"
I20260928 06:28:57.487560 31953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:28:57.495841 31953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:28:57.498147 31953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:28:57.498299 31953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:28:57.498520 31953 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:28:57.498847 31953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:28:57.498919 31953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:57.498961 31953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:28:57.499001 31953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:57.504875 31953 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.131:43215
I20260928 06:28:57.504943 32081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.131:43215 every 8 connection(s)
I20260928 06:28:57.505246 31953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
I20260928 06:28:57.509089 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 31953
I20260928 06:28:57.509195 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-2/wal/instance
I20260928 06:28:57.509394 23430 external_mini_cluster.cc:1428] Setting key db4ab1f8408459605e9dadb9a846d2c4
Sep 28 06:28:57 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576937, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.131@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:28:57.515447 31679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:57.505975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:52641 (local address 127.22.225.190:40039)
0928 06:28:57.506047 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:57.506049 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:57.506272 (+ 223us) server_negotiation.cc:407] Connection header received
0928 06:28:57.506872 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:57.506873 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:57.506883 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:57.506935 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:57.508101 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:57.508645 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:57.510110 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:57.510388 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:57.510440 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:57.512756 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:57.512758 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:57.512758 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:57.512765 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:57.514147 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:57.514551 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:57.514552 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:57.514553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:57.514621 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:57.514849 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:57.514849 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:57.514850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:57.514978 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:57.515046 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:57.515272 (+ 226us) server_negotiation.cc:299] Negotiation successful
0928 06:28:57.515349 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 06:28:57.515537 32084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:57.505944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:40039 (local address 127.22.225.131:52641)
0928 06:28:57.506149 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:57.506194 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:57.506773 (+ 579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:57.507009 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:57.507013 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:57.507752 (+ 739us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:57.508006 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:57.508013 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:57.508733 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:57.508734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:57.509885 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:57.509888 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:57.510465 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:57.510466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:57.510612 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:57.511098 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:57.511107 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:57.512643 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:57.514242 (+ 1599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:57.514248 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:57.514249 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:57.514472 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:57.514691 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:57.514691 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:57.514692 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:57.514785 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:57.515041 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:57.515045 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:57.515149 (+ 104us) client_negotiation.cc:769] Sending connection context
0928 06:28:57.515229 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 06:28:57.515343 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":109,"threads_started":1}
I20260928 06:28:57.516048 32082 heartbeater.cc:347] Connected to a master server at 127.22.225.190:40039
I20260928 06:28:57.516155 32082 heartbeater.cc:464] Registering TS with master...
I20260928 06:28:57.516348 32082 heartbeater.cc:511] Master 127.22.225.190:40039 requested a full tablet report, sending...
I20260928 06:28:57.516798 31623 ts_manager.cc:194] Registered new tserver with Master: 4ee0d29c8a414813a2bd0adbfa0a0350 (127.22.225.131:43215)
I20260928 06:28:57.517174 31623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.225.131@KRBTEST.COM'} at 127.22.225.131:52641
I20260928 06:28:57.520967 23430 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.22.225.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.225.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.225.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/krb5kdc/oryx_127.22.225.129.keytab.
Entry for principal oryx/127.22.225.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/krb5kdc/oryx_127.22.225.129.keytab.
WARNING: no policy specified for oryx/127.22.225.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.225.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.225.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/krb5kdc/oryx_127.22.225.130.keytab.
Entry for principal oryx/127.22.225.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/krb5kdc/oryx_127.22.225.130.keytab.
WARNING: no policy specified for oryx/127.22.225.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.225.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.225.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/krb5kdc/oryx_127.22.225.131.keytab.
Entry for principal oryx/127.22.225.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/krb5kdc/oryx_127.22.225.131.keytab.
I20260928 06:28:57.591636 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 31685
I20260928 06:28:57.597119 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 31819
I20260928 06:28:57.602775 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 31953
I20260928 06:28:57.607285 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 31591
I20260928 06:28:57.612941 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:40039
--webserver_interface=127.22.225.190
--webserver_port=40741
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab
--principal=kudu/127.22.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:36285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.225.190:40039
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:28:57.694020 32091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:28:57.694299 32091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:28:57.694322 32091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:28:57.695485 32091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:28:57.695529 32091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:28:57.695544 32091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:28:57.695556 32091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:28:57.695568 32091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:28:57.697196 32091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:36285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.225.190:40039
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab
--principal=kudu/127.22.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:40039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.225.190
--webserver_port=40741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:28:57.697458 32091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:28:57.697693 32091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:28:57.702410 32097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:57.702404 32096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:57.702607 32091 server_base.cc:1034] running on GCE node
W20260928 06:28:57.702709 32099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:57.702971 32091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:28:57.703295 32091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:28:57.704461 32091 hybrid_clock.cc:648] HybridClock initialized: now 1790576937704442 us; error 40 us; skew 500 ppm
Sep 28 06:28:57 dist-test-slave-d59m krb5kdc[31578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576937, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:28:57.707083 32091 init.cc:377] Logged in from keytab as kudu/127.22.225.190@KRBTEST.COM (short username kudu)
I20260928 06:28:57.707541 32091 webserver.cc:460] Webserver started at http://127.22.225.190:40741/ using document root <none> and password file <none>
I20260928 06:28:57.707794 32091 fs_manager.cc:359] Metadata directory not provided
I20260928 06:28:57.707880 32091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:28:57.709316 32091 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:28:57.710019 32106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:57.710193 32091 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:28:57.710290 32091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/master-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/master-0/wal
uuid: "c94b06e2757140948bc74d9b1dd41cf1"
format_stamp: "Formatted at 2026-09-28 06:28:56 on dist-test-slave-d59m"
server_key: "4691bb3888f496c359e6b19776ab85a9"
server_key_iv: "345c8a2c06e26cf9c76f1876d0d70d6f"
server_key_version: "encryptionkey@0"
I20260928 06:28:57.710642 32091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:28:57.719236 32091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:28:57.721553 32091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:28:57.721710 32091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:28:57.726274 32091 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.190:40039
I20260928 06:28:57.726298 32158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.190:40039 every 8 connection(s)
I20260928 06:28:57.726670 32091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
I20260928 06:28:57.727346 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 32091
I20260928 06:28:57.727659 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.129:33011
--local_ip_for_outbound_sockets=127.22.225.129
--tserver_master_addrs=127.22.225.190:40039
--webserver_port=46341
--webserver_interface=127.22.225.129
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab
--principal=kudu/127.22.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:36285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.22.225.129
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/krb5kdc/oryx_127.22.225.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:28:57.728291 32159 sys_catalog.cc:263] Verifying existing consensus state
I20260928 06:28:57.729202 32159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c94b06e2757140948bc74d9b1dd41cf1: Bootstrap starting.
I20260928 06:28:57.737385 32159 log.cc:824] T 00000000000000000000000000000000 P c94b06e2757140948bc74d9b1dd41cf1: Log is configured to *not* fsync() on all Append() calls
I20260928 06:28:57.738961 32159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c94b06e2757140948bc74d9b1dd41cf1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:28:57.739250 32159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c94b06e2757140948bc74d9b1dd41cf1: Bootstrap complete.
I20260928 06:28:57.741642 32159 raft_consensus.cc:348] T 00000000000000000000000000000000 P c94b06e2757140948bc74d9b1dd41cf1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c94b06e2757140948bc74d9b1dd41cf1" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 40039 } }
I20260928 06:28:57.741882 32159 raft_consensus.cc:729] T 00000000000000000000000000000000 P c94b06e2757140948bc74d9b1dd41cf1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c94b06e2757140948bc74d9b1dd41cf1, State: Initialized, Role: FOLLOWER
I20260928 06:28:57.742091 32159 consensus_queue.cc:260] T 00000000000000000000000000000000 P c94b06e2757140948bc74d9b1dd41cf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c94b06e2757140948bc74d9b1dd41cf1" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 40039 } }
I20260928 06:28:57.742187 32159 raft_consensus.cc:388] T 00000000000000000000000000000000 P c94b06e2757140948bc74d9b1dd41cf1 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:28:57.742249 32159 raft_consensus.cc:482] T 00000000000000000000000000000000 P c94b06e2757140948bc74d9b1dd41cf1 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:28:57.742304 32159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c94b06e2757140948bc74d9b1dd41cf1 [term 1 FOLLOWER]: Advancing to term 2
I20260928 06:28:57.743238 32159 raft_consensus.cc:504] T 00000000000000000000000000000000 P c94b06e2757140948bc74d9b1dd41cf1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c94b06e2757140948bc74d9b1dd41cf1" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 40039 } }
I20260928 06:28:57.743362 32159 leader_election.cc:302] T 00000000000000000000000000000000 P c94b06e2757140948bc74d9b1dd41cf1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c94b06e2757140948bc74d9b1dd41cf1; no voters:
I20260928 06:28:57.743537 32159 leader_election.cc:288] T 00000000000000000000000000000000 P c94b06e2757140948bc74d9b1dd41cf1 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 06:28:57.743573 32163 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c94b06e2757140948bc74d9b1dd41cf1 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 06:28:57.743768 32163 raft_consensus.cc:686] T 00000000000000000000000000000000 P c94b06e2757140948bc74d9b1dd41cf1 [term 2 LEADER]: Becoming Leader. State: Replica: c94b06e2757140948bc74d9b1dd41cf1, State: Running, Role: LEADER
I20260928 06:28:57.743877 32159 sys_catalog.cc:564] T 00000000000000000000000000000000 P c94b06e2757140948bc74d9b1dd41cf1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:28:57.743901 32163 consensus_queue.cc:237] T 00000000000000000000000000000000 P c94b06e2757140948bc74d9b1dd41cf1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c94b06e2757140948bc74d9b1dd41cf1" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 40039 } }
I20260928 06:28:57.744223 32163 sys_catalog.cc:455] T 00000000000000000000000000000000 P c94b06e2757140948bc74d9b1dd41cf1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c94b06e2757140948bc74d9b1dd41cf1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c94b06e2757140948bc74d9b1dd41cf1" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 40039 } } }
I20260928 06:28:57.744369 32163 sys_catalog.cc:458] T 00000000000000000000000000000000 P c94b06e2757140948bc74d9b1dd41cf1 [sys.catalog]: This master's current role is: LEADER
I20260928 06:28:57.744546 32163 sys_catalog.cc:455] T 00000000000000000000000000000000 P c94b06e2757140948bc74d9b1dd41cf1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c94b06e2757140948bc74d9b1dd41cf1. Latest consensus state: current_term: 2 leader_uuid: "c94b06e2757140948bc74d9b1dd41cf1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c94b06e2757140948bc74d9b1dd41cf1" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 40039 } } }
I20260928 06:28:57.744694 32163 sys_catalog.cc:458] T 00000000000000000000000000000000 P c94b06e2757140948bc74d9b1dd41cf1 [sys.catalog]: This master's current role is: LEADER
I20260928 06:28:57.745945 32178 catalog_manager.cc:1259] Loaded cluster ID: 14e64f749762418ea449f4046d5278d5
I20260928 06:28:57.746006 32178 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c94b06e2757140948bc74d9b1dd41cf1: loading cluster ID for follower catalog manager: success
I20260928 06:28:57.746833 32178 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c94b06e2757140948bc74d9b1dd41cf1: acquiring CA information for follower catalog manager: success
I20260928 06:28:57.747126 32178 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c94b06e2757140948bc74d9b1dd41cf1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 06:28:57.747210 32171 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:28:57.747349 32171 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:28:57.747426 32171 catalog_manager.cc:1259] Loaded cluster ID: 14e64f749762418ea449f4046d5278d5
I20260928 06:28:57.747457 32171 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:28:57.747623 32171 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:28:57.747715 32171 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c94b06e2757140948bc74d9b1dd41cf1: Loaded TSK: 0
I20260928 06:28:57.747851 32171 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 06:28:57.842039 32161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:28:57.842316 32161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:28:57.842350 32161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:28:57.843921 32161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:28:57.843976 32161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:28:57.844012 32161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.129
I20260928 06:28:57.845948 32161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:36285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/krb5kdc/oryx_127.22.225.129.keytab
--principal=oryx/127.22.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.129:33011
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.225.129
--webserver_port=46341
--tserver_master_addrs=127.22.225.190:40039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.129
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:28:57.846176 32161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:28:57.846423 32161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:28:57.852540 32186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:57.852547 32188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:57.852924 32161 server_base.cc:1034] running on GCE node
W20260928 06:28:57.853022 32185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:57.853223 32161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:28:57.853477 32161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:28:57.854640 32161 hybrid_clock.cc:648] HybridClock initialized: now 1790576937854612 us; error 39 us; skew 500 ppm
Sep 28 06:28:57 dist-test-slave-d59m krb5kdc[31578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576937, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:28:57.857700 32161 init.cc:377] Logged in from keytab as oryx/127.22.225.129@KRBTEST.COM (short username oryx)
I20260928 06:28:57.858218 32161 webserver.cc:460] Webserver started at http://127.22.225.129:46341/ using document root <none> and password file <none>
I20260928 06:28:57.858472 32161 fs_manager.cc:359] Metadata directory not provided
I20260928 06:28:57.858543 32161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:28:57.860285 32161 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:28:57.861006 32195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:57.861148 32161 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:28:57.861222 32161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-0/wal
uuid: "9a5de9d8fe35416ea912d2ab0da02a45"
format_stamp: "Formatted at 2026-09-28 06:28:57 on dist-test-slave-d59m"
server_key: "25d7ab184bc26929490e3862dbc4896d"
server_key_iv: "0ff531a35cc5fb5c00fa36a229e146ee"
server_key_version: "encryptionkey@0"
I20260928 06:28:57.861650 32161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:28:57.886404 32161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:28:57.888906 32161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:28:57.889077 32161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:28:57.889350 32161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:28:57.889668 32161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:28:57.889722 32161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:57.889766 32161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:28:57.889791 32161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:57.895385 32161 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.129:33011
I20260928 06:28:57.895494 32308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.129:33011 every 8 connection(s)
I20260928 06:28:57.895830 32161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
Sep 28 06:28:57 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.129@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:28:57 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.129@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:28:57.903961 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 32161
I20260928 06:28:57.904273 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.130:33687
--local_ip_for_outbound_sockets=127.22.225.130
--tserver_master_addrs=127.22.225.190:40039
--webserver_port=32957
--webserver_interface=127.22.225.130
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab
--principal=kudu/127.22.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:36285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.22.225.130
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/krb5kdc/oryx_127.22.225.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:28:57.904390 32311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:57.896565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:40039 (local address 127.22.225.129:48037)
0928 06:28:57.896778 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:57.896824 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:57.897286 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:57.897735 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:57.897740 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:57.898296 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:57.898480 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:57.898485 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:57.899725 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:57.899726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:57.900504 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:57.900506 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:57.901157 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:57.901158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:57.901287 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:57.901801 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:57.901809 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:57.904186 (+ 2377us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.225.190:40039: Server oryx/127.22.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":90,"threads_started":1}
W20260928 06:28:57.904790 32309 heartbeater.cc:644] Failed to heartbeat to 127.22.225.190:40039 (0 consecutive failures): Not authorized: Failed to ping master at 127.22.225.190:40039: Client connection negotiation failed: client connection to 127.22.225.190:40039: Server oryx/127.22.225.190@KRBTEST.COM not found in Kerberos database
W20260928 06:28:57.905256 32312 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:28:57.896645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:48037 (local address 127.22.225.190:40039)
0928 06:28:57.896852 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:57.896872 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:57.896905 (+ 33us) server_negotiation.cc:407] Connection header received
0928 06:28:57.897507 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:57.897511 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:57.897553 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:57.897683 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:57.898605 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:57.899617 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:57.900644 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:57.901053 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:57.901181 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:57.905090 (+ 3909us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.225.129:48037: BlockingRecv error: recv got EOF from 127.22.225.129:48037 (error 108)
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":105,"threads_started":1}
Sep 28 06:28:57 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.129@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:28:57 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.129@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:28:57.910223 32311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:57.905023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:40039 (local address 127.22.225.129:57865)
0928 06:28:57.905377 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:57.905388 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:57.905431 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:57.905763 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:57.905764 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:57.905906 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:57.905947 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:57.905948 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:57.906594 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:57.906595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:57.907096 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:57.907097 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:57.907562 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:57.907563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:57.907637 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:57.908108 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:57.908113 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:57.910130 (+ 2017us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.225.190:40039: Server oryx/127.22.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":338}
W20260928 06:28:57.910471 32312 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:28:57.905571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:57865 (local address 127.22.225.190:40039)
0928 06:28:57.905634 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:57.905636 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:57.905648 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:28:57.905671 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:57.905672 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:57.905683 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:57.905759 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:57.906030 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:57.906510 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:57.907180 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:57.907477 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:57.907548 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:57.910401 (+ 2853us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.225.129:57865: BlockingRecv error: recv got EOF from 127.22.225.129:57865 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 28 06:28:57 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.129@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:28:57 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.129@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:28:57.915968 32311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:57.910490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:40039 (local address 127.22.225.129:49311)
0928 06:28:57.910813 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:57.910824 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:57.910878 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:57.911026 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:57.911027 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:57.911165 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:57.911198 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:57.911200 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:57.912141 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:57.912142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:57.912961 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:57.912963 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:57.913495 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:57.913496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:57.913542 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:57.914030 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:57.914035 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:57.915912 (+ 1877us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.225.190:40039: Server oryx/127.22.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":308}
W20260928 06:28:57.916114 32312 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:28:57.910573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:49311 (local address 127.22.225.190:40039)
0928 06:28:57.910607 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:57.910609 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:57.910886 (+ 277us) server_negotiation.cc:407] Connection header received
0928 06:28:57.910922 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:57.910922 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:57.910936 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:57.911012 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:57.911278 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:57.912047 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:57.913046 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:57.913407 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:57.913485 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:57.916034 (+ 2549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.225.129:49311: BlockingRecv error: recv got EOF from 127.22.225.129:49311 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260928 06:28:57.916301 32309 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 06:28:57.989874 32313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:28:57.990079 32313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:28:57.990103 32313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:28:57.991272 32313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:28:57.991313 32313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:28:57.991339 32313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.130
I20260928 06:28:57.992839 32313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:36285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/krb5kdc/oryx_127.22.225.130.keytab
--principal=oryx/127.22.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.130:33687
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.225.130
--webserver_port=32957
--tserver_master_addrs=127.22.225.190:40039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.130
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:28:57.993088 32313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:28:57.993388 32313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:28:57.998271 32319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:57.998486 32321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:57.998425 32318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:57.998867 32313 server_base.cc:1034] running on GCE node
I20260928 06:28:57.999105 32313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:28:57.999341 32313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:28:58.000485 32313 hybrid_clock.cc:648] HybridClock initialized: now 1790576938000473 us; error 28 us; skew 500 ppm
Sep 28 06:28:57 dist-test-slave-d59m krb5kdc[31578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576937, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:28:58.002981 32313 init.cc:377] Logged in from keytab as oryx/127.22.225.130@KRBTEST.COM (short username oryx)
I20260928 06:28:58.003464 32313 webserver.cc:460] Webserver started at http://127.22.225.130:32957/ using document root <none> and password file <none>
I20260928 06:28:58.003705 32313 fs_manager.cc:359] Metadata directory not provided
I20260928 06:28:58.003760 32313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:28:58.005224 32313 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:28:58.005868 32328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:58.006084 32313 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:28:58.006196 32313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-1/wal
uuid: "2186c977b9be4d8a9c16d6bdf139d872"
format_stamp: "Formatted at 2026-09-28 06:28:57 on dist-test-slave-d59m"
server_key: "74bbeedeaaff13f9c1eb7855d9a98a93"
server_key_iv: "87d31fc7fd6edce503f89ddf9b96103c"
server_key_version: "encryptionkey@0"
I20260928 06:28:58.006552 32313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:28:58.017292 32313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:28:58.019541 32313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:28:58.019704 32313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:28:58.019958 32313 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:28:58.020274 32313 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:28:58.020329 32313 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:58.020366 32313 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:28:58.020393 32313 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:58.026563 32313 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.130:33687
I20260928 06:28:58.026666 32441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.130:33687 every 8 connection(s)
I20260928 06:28:58.026974 32313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
I20260928 06:28:58.030227 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 32313
I20260928 06:28:58.030525 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.131:43215
--local_ip_for_outbound_sockets=127.22.225.131
--tserver_master_addrs=127.22.225.190:40039
--webserver_port=33137
--webserver_interface=127.22.225.131
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab
--principal=kudu/127.22.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:36285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.22.225.131
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/krb5kdc/oryx_127.22.225.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 06:28:58 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.130@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:28:58 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.130@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:28:58.036365 32444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:58.027768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:40039 (local address 127.22.225.130:46607)
0928 06:28:58.027955 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:58.028001 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:58.028664 (+ 663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:58.028914 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:58.028918 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:58.029668 (+ 750us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:58.029933 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:58.029940 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:58.030942 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:58.030943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:58.032114 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:58.032117 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:58.032651 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:58.032653 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:58.032786 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:58.033395 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:58.033404 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:58.036159 (+ 2755us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.225.190:40039: Server oryx/127.22.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":90,"threads_started":1}
W20260928 06:28:58.036374 32312 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:28:58.027801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:46607 (local address 127.22.225.190:40039)
0928 06:28:58.027858 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:58.027860 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:58.028104 (+ 244us) server_negotiation.cc:407] Connection header received
0928 06:28:58.028780 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:58.028781 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:58.028796 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:58.028839 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:58.030069 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:58.030831 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:58.032214 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:58.032541 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:58.032643 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:58.036285 (+ 3642us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.225.130:46607: BlockingRecv error: recv got EOF from 127.22.225.130:46607 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260928 06:28:58.036666 32442 heartbeater.cc:644] Failed to heartbeat to 127.22.225.190:40039 (0 consecutive failures): Not authorized: Failed to ping master at 127.22.225.190:40039: Client connection negotiation failed: client connection to 127.22.225.190:40039: Server oryx/127.22.225.190@KRBTEST.COM not found in Kerberos database
Sep 28 06:28:58 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.130@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:28:58 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.130@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:28:58.042248 32444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:58.036913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:40039 (local address 127.22.225.130:37877)
0928 06:28:58.036960 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:58.036971 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:58.037009 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:58.037159 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:58.037160 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:58.037287 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:58.037336 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:58.037338 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:58.038317 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:58.038318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:58.038986 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:58.038988 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:58.039523 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:58.039524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:58.039585 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:58.040066 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:58.040072 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:58.042186 (+ 2114us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.225.190:40039: Server oryx/127.22.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 06:28:58.042439 32312 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:28:58.036951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:37877 (local address 127.22.225.190:40039)
0928 06:28:58.037025 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:58.037027 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:58.037037 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:28:58.037066 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:58.037067 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:58.037078 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:58.037146 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:58.037427 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:58.038225 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:58.039081 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:58.039416 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:58.039512 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:58.042296 (+ 2784us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.225.130:37877: BlockingRecv error: recv got EOF from 127.22.225.130:37877 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 28 06:28:58 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.130@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:28:58 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.130@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:28:58.047449 32444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:58.042589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:40039 (local address 127.22.225.130:55103)
0928 06:28:58.042626 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:58.042636 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:58.042680 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:58.042816 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:58.042817 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:58.042934 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:58.042962 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:58.042963 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:58.043860 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:58.043861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:58.044536 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:58.044539 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:58.045099 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:58.045100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:58.045147 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:58.045581 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:58.045586 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:58.047395 (+ 1809us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.225.190:40039: Server oryx/127.22.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":24}
W20260928 06:28:58.047546 32442 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 06:28:58.047564 32312 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:28:58.042597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:55103 (local address 127.22.225.190:40039)
0928 06:28:58.042627 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:58.042629 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:58.042682 (+ 53us) server_negotiation.cc:407] Connection header received
0928 06:28:58.042740 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:58.042741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:58.042749 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:58.042799 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:58.043032 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:58.043771 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:58.044715 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:58.045033 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:58.045099 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:58.047493 (+ 2394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.225.130:55103: BlockingRecv error: recv got EOF from 127.22.225.130:55103 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
W20260928 06:28:58.117624 32445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:28:58.117830 32445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:28:58.117866 32445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:28:58.118995 32445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:28:58.119057 32445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:28:58.119102 32445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.131
I20260928 06:28:58.120504 32445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:36285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/krb5kdc/oryx_127.22.225.131.keytab
--principal=oryx/127.22.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.131:43215
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.225.131
--webserver_port=33137
--tserver_master_addrs=127.22.225.190:40039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.131
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:28:58.120786 32445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:28:58.121011 32445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:28:58.126189 32450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:58.126196 32453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:58.126205 32451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:58.126390 32445 server_base.cc:1034] running on GCE node
I20260928 06:28:58.126591 32445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:28:58.126847 32445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:28:58.128005 32445 hybrid_clock.cc:648] HybridClock initialized: now 1790576938127992 us; error 38 us; skew 500 ppm
Sep 28 06:28:58 dist-test-slave-d59m krb5kdc[31578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576938, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:28:58.130431 32445 init.cc:377] Logged in from keytab as oryx/127.22.225.131@KRBTEST.COM (short username oryx)
I20260928 06:28:58.130913 32445 webserver.cc:460] Webserver started at http://127.22.225.131:33137/ using document root <none> and password file <none>
I20260928 06:28:58.131162 32445 fs_manager.cc:359] Metadata directory not provided
I20260928 06:28:58.131217 32445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:28:58.132660 32445 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:28:58.133297 32460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:58.133464 32445 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:28:58.133558 32445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-2/wal
uuid: "4ee0d29c8a414813a2bd0adbfa0a0350"
format_stamp: "Formatted at 2026-09-28 06:28:57 on dist-test-slave-d59m"
server_key: "f1609bd26aae734a74b78793826cf8ee"
server_key_iv: "2ce638761fc986a0731976a190a2989c"
server_key_version: "encryptionkey@0"
I20260928 06:28:58.133862 32445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:28:58.155731 32445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:28:58.158282 32445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:28:58.158430 32445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:28:58.158707 32445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:28:58.159040 32445 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:28:58.159073 32445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:58.159097 32445 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:28:58.159137 32445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:58.165441 32445 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.131:43215
I20260928 06:28:58.165491 32573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.131:43215 every 8 connection(s)
I20260928 06:28:58.165891 32445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
Sep 28 06:28:58 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.131@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:28:58 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.131@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
W20260928 06:28:58.173739 32312 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:28:58.166722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:59175 (local address 127.22.225.190:40039)
0928 06:28:58.166791 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:58.166794 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:58.166970 (+ 176us) server_negotiation.cc:407] Connection header received
0928 06:28:58.167457 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:58.167459 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:58.167471 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:58.167567 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:58.168460 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:58.169274 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:58.170240 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:58.170550 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:58.170626 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:58.173692 (+ 3066us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.225.131:59175: BlockingRecv error: recv got EOF from 127.22.225.131:59175 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 06:28:58.173893 32576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:58.166695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:40039 (local address 127.22.225.131:59175)
0928 06:28:58.166842 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:58.166884 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:58.167360 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:58.167619 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:58.167622 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:58.168176 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:58.168365 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:58.168369 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:58.169363 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:58.169364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:58.170153 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:58.170155 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:58.170635 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:58.170636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:58.170729 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:58.171265 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:58.171273 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:58.173620 (+ 2347us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.225.190:40039: Server oryx/127.22.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":77,"threads_started":1}
W20260928 06:28:58.174149 32574 heartbeater.cc:644] Failed to heartbeat to 127.22.225.190:40039 (0 consecutive failures): Not authorized: Failed to ping master at 127.22.225.190:40039: Client connection negotiation failed: client connection to 127.22.225.190:40039: Server oryx/127.22.225.190@KRBTEST.COM not found in Kerberos database
I20260928 06:28:58.175941 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 32445
Sep 28 06:28:58 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.131@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:28:58 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.131@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:28:58.178881 32576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:58.174365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:40039 (local address 127.22.225.131:40757)
0928 06:28:58.174416 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:58.174429 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:58.174499 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:58.174686 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:58.174687 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:58.174817 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:58.174857 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:58.174859 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:58.175485 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:58.175486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:58.176037 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:58.176039 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:58.176486 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:58.176487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:58.176561 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:58.177042 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:58.177048 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:58.178802 (+ 1754us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.225.190:40039: Server oryx/127.22.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 06:28:58.178930 32312 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:28:58.174536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:40757 (local address 127.22.225.190:40039)
0928 06:28:58.174570 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:58.174572 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:58.174578 (+ 6us) server_negotiation.cc:407] Connection header received
0928 06:28:58.174597 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:58.174598 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:58.174606 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:58.174682 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:58.174931 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:58.175393 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:58.176105 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:58.176394 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:58.176729 (+ 335us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:58.178869 (+ 2140us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.225.131:40757: BlockingRecv error: recv got EOF from 127.22.225.131:40757 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Sep 28 06:28:58 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.131@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:28:58 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.131@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:28:58.183866 32576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:58.179145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:40039 (local address 127.22.225.131:58283)
0928 06:28:58.179277 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:58.179287 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:58.179331 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:58.179456 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:58.179457 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:58.179579 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:58.179602 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:58.179603 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:58.180382 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:58.180383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:58.181275 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:58.181278 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:58.181622 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:58.181623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:58.181674 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:58.182181 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:58.182185 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:58.183784 (+ 1599us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.225.190:40039: Server oryx/127.22.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118}
W20260928 06:28:58.183915 32312 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:28:58.179215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:58283 (local address 127.22.225.190:40039)
0928 06:28:58.179277 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:58.179279 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:58.179318 (+ 39us) server_negotiation.cc:407] Connection header received
0928 06:28:58.179387 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:58.179388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:58.179397 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:58.179458 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:58.179683 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:58.180277 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:58.181360 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:58.181549 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:58.181625 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:58.183851 (+ 2226us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.225.131:58283: BlockingRecv error: recv got EOF from 127.22.225.131:58283 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260928 06:28:58.184335 32574 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 06:28:58.185385 32578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:58.176329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33506 (local address 127.22.225.190:40039)
0928 06:28:58.176461 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:58.176463 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:58.176510 (+ 47us) server_negotiation.cc:407] Connection header received
0928 06:28:58.176566 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:58.176566 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:58.176604 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:58.176671 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:58.177403 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:58.178158 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:58.179417 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:58.179796 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:58.179865 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:58.181364 (+ 1499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:58.181371 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:58.181371 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:58.181388 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:58.183737 (+ 2349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:58.184211 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:58.184214 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:58.184215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:58.184358 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:58.184631 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:58.184632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:58.184632 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:58.184882 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:58.185039 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:58.185160 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 06:28:58.185277 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":61,"threads_started":1}
Sep 28 06:28:58 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.129@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:28:58 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.129@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:28:58.928340 32579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:58.916742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:40039 (local address 127.22.225.129:52619)
0928 06:28:58.916869 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:58.916882 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:58.916946 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:58.917098 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:58.917098 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:58.917285 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:58.917333 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:58.917334 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:58.918126 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:58.918127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:58.918684 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:58.918685 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:58.919111 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:58.919112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:58.919150 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:58.926430 (+ 7280us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:58.926436 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:58.928273 (+ 1837us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.225.190:40039: Server oryx/127.22.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":66,"threads_started":1}
W20260928 06:28:58.928463 32580 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:28:58.916818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:52619 (local address 127.22.225.190:40039)
0928 06:28:58.916958 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:58.916960 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:58.916969 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:28:58.917002 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:58.917002 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:58.917014 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:58.917085 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:58.917406 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:58.918039 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:58.918779 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:58.919030 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:58.919112 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:58.928396 (+ 9284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.225.129:52619: BlockingRecv error: recv got EOF from 127.22.225.129:52619 (error 108)
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":70,"threads_started":1}
Sep 28 06:28:59 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.130@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:28:59 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.130@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:28:59.052954 32581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:59.047946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:40039 (local address 127.22.225.130:58333)
0928 06:28:59.048106 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:59.048118 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:59.048188 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:59.048332 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:59.048333 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:59.048533 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:59.048626 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:59.048629 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:59.049281 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:59.049283 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:59.050064 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:59.050067 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:59.050410 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:59.050411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:59.050465 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:59.050904 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:59.050909 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:59.052868 (+ 1959us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.225.190:40039: Server oryx/127.22.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":80,"threads_started":1}
W20260928 06:28:59.053040 32580 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:28:59.047998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:58333 (local address 127.22.225.190:40039)
0928 06:28:59.048071 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:59.048073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:59.048186 (+ 113us) server_negotiation.cc:407] Connection header received
0928 06:28:59.048249 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:59.048250 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:59.048260 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:59.048311 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:59.048690 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:59.049207 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:59.050152 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:59.050338 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:59.050391 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:59.052963 (+ 2572us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.225.130:58333: BlockingRecv error: recv got EOF from 127.22.225.130:58333 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 28 06:28:59 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.131@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:28:59 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.131@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:28:59.189932 32582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:59.184770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:40039 (local address 127.22.225.131:46471)
0928 06:28:59.184937 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:59.184947 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:59.185005 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:59.185177 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:59.185178 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:59.185341 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:59.185426 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:59.185428 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:59.186352 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:59.186353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:59.187001 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:59.187003 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:59.187451 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:59.187452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:59.187491 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:59.188004 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:59.188009 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:59.189864 (+ 1855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.225.190:40039: Server oryx/127.22.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":86,"threads_started":1}
W20260928 06:28:59.190037 32580 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:28:59.184861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:46471 (local address 127.22.225.190:40039)
0928 06:28:59.184920 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:59.184922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:59.184996 (+ 74us) server_negotiation.cc:407] Connection header received
0928 06:28:59.185052 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:59.185053 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:59.185068 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:59.185138 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:59.185542 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:59.186260 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:59.187095 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:59.187370 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:59.187447 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:59.189976 (+ 2529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.225.131:46471: BlockingRecv error: recv got EOF from 127.22.225.131:46471 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 28 06:28:59 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.129@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:28:59 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.129@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:28:59.933995 32583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:59.928816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:40039 (local address 127.22.225.129:59555)
0928 06:28:59.928942 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:59.928956 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:59.929012 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:59.929153 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:59.929154 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:59.929361 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:59.929426 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:59.929428 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:59.930071 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:59.930072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:59.930921 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:59.930925 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:59.931267 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:59.931269 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:59.931334 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:59.931800 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:59.931806 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:59.933891 (+ 2085us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.225.190:40039: Server oryx/127.22.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":78,"threads_started":1}
W20260928 06:28:59.934043 32584 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:28:59.928854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:59555 (local address 127.22.225.190:40039)
0928 06:28:59.928970 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:59.928972 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:59.928993 (+ 21us) server_negotiation.cc:407] Connection header received
0928 06:28:59.929076 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:59.929076 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:59.929086 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:59.929146 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:59.929504 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:59.929986 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:59.931014 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:59.931198 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:59.931243 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:59.933973 (+ 2730us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.225.129:59555: BlockingRecv error: recv got EOF from 127.22.225.129:59555 (error 108)
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":71,"threads_started":1}
Sep 28 06:29:00 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.130@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:29:00 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.130@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:29:00.058205 32585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:00.053381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:40039 (local address 127.22.225.130:43755)
0928 06:29:00.053535 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 06:29:00.053546 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:29:00.053617 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:29:00.053786 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:29:00.053787 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:29:00.053961 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:29:00.054016 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:00.054018 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:00.054721 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:00.054722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:00.055253 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:00.055254 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:00.055630 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:00.055631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:00.055687 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:00.056196 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:29:00.056201 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:29:00.058112 (+ 1911us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.225.190:40039: Server oryx/127.22.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":69,"threads_started":1}
W20260928 06:29:00.058394 32584 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:29:00.053461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:43755 (local address 127.22.225.190:40039)
0928 06:29:00.053540 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 06:29:00.053543 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:29:00.053587 (+ 44us) server_negotiation.cc:407] Connection header received
0928 06:29:00.053680 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:29:00.053681 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:29:00.053700 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:29:00.053782 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:29:00.054104 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:00.054625 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:00.055339 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:00.055541 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:00.055631 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:00.058249 (+ 2618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.225.130:43755: BlockingRecv error: recv got EOF from 127.22.225.130:43755 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 28 06:29:00 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.131@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:29:00 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.131@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:29:00.195659 32586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:00.190631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:40039 (local address 127.22.225.131:32889)
0928 06:29:00.190771 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 06:29:00.190784 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:29:00.190860 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:29:00.190998 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:29:00.190999 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:29:00.191238 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:29:00.191301 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:00.191303 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:00.191931 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:00.191932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:00.192702 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:00.192706 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:00.193077 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:00.193078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:00.193132 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:00.193564 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:29:00.193570 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:29:00.195562 (+ 1992us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.225.190:40039: Server oryx/127.22.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":70,"threads_started":1}
W20260928 06:29:00.195711 32584 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:29:00.190749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:32889 (local address 127.22.225.190:40039)
0928 06:29:00.190794 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 06:29:00.190796 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:29:00.190843 (+ 47us) server_negotiation.cc:407] Connection header received
0928 06:29:00.190904 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:29:00.190904 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:29:00.190915 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:29:00.190965 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:29:00.191383 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:00.191854 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:00.192790 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:00.192998 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:00.193055 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:00.195646 (+ 2591us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.225.131:32889: BlockingRecv error: recv got EOF from 127.22.225.131:32889 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 28 06:29:00 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.129@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:29:00 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.129@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:29:00.939724 32587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:00.934424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:40039 (local address 127.22.225.129:47637)
0928 06:29:00.934625 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 06:29:00.934635 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:29:00.934689 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:29:00.934836 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:29:00.934836 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:29:00.934992 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:29:00.935047 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:00.935048 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:00.936071 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:00.936072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:00.936596 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:00.936598 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:00.937128 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:00.937129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:00.937173 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:00.937672 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:29:00.937677 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:29:00.939607 (+ 1930us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.225.190:40039: Server oryx/127.22.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":95,"threads_started":1}
W20260928 06:29:00.939750 32588 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:29:00.934508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:47637 (local address 127.22.225.190:40039)
0928 06:29:00.934634 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 06:29:00.934636 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:29:00.934677 (+ 41us) server_negotiation.cc:407] Connection header received
0928 06:29:00.934736 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:29:00.934737 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:29:00.934751 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:29:00.934825 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:29:00.935119 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:00.935992 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:00.936691 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:00.937049 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:00.937130 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:00.939696 (+ 2566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.225.129:47637: BlockingRecv error: recv got EOF from 127.22.225.129:47637 (error 108)
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":53,"threads_started":1}
Sep 28 06:29:01 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.130@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:29:01 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.130@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:29:01.063576 32589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:01.058647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:40039 (local address 127.22.225.130:38623)
0928 06:29:01.058819 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 06:29:01.058828 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:29:01.058879 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:29:01.059049 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:29:01.059050 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:29:01.059200 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:29:01.059245 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:01.059247 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:01.060099 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:01.060100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:01.060661 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:01.060663 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:01.061098 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:01.061099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:01.061137 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:01.061635 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:29:01.061640 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:29:01.063500 (+ 1860us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.225.190:40039: Server oryx/127.22.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":92,"threads_started":1}
W20260928 06:29:01.063691 32588 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:29:01.058753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:38623 (local address 127.22.225.190:40039)
0928 06:29:01.058818 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 06:29:01.058820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:29:01.058882 (+ 62us) server_negotiation.cc:407] Connection header received
0928 06:29:01.058960 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:29:01.058961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:29:01.058973 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:29:01.059027 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:29:01.059311 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:01.060018 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:01.060733 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:01.061016 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:01.061089 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:01.063593 (+ 2504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.225.130:38623: BlockingRecv error: recv got EOF from 127.22.225.130:38623 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 28 06:29:01 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.131@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:29:01 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.131@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:29:01.201141 32590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:01.196192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:40039 (local address 127.22.225.131:45105)
0928 06:29:01.196359 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 06:29:01.196371 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:29:01.196431 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:29:01.196676 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:29:01.196677 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:29:01.196846 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:29:01.196893 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:01.196894 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:01.197721 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:01.197721 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:01.198275 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:01.198277 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:01.198721 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:01.198721 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:01.198762 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:01.199244 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:29:01.199249 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:29:01.201083 (+ 1834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.225.190:40039: Server oryx/127.22.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":98,"threads_started":1}
W20260928 06:29:01.201259 32588 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:29:01.196319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:45105 (local address 127.22.225.190:40039)
0928 06:29:01.196393 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 06:29:01.196395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:29:01.196422 (+ 27us) server_negotiation.cc:407] Connection header received
0928 06:29:01.196495 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:29:01.196496 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:29:01.196512 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:29:01.196612 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:29:01.196974 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:01.197638 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:01.198368 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:01.198634 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:01.198715 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:01.201198 (+ 2483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.225.131:45105: BlockingRecv error: recv got EOF from 127.22.225.131:45105 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 28 06:29:01 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.129@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:29:01 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.129@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:29:01.945468 32591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:01.940344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:40039 (local address 127.22.225.129:35415)
0928 06:29:01.940470 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 06:29:01.940482 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:29:01.940557 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:29:01.940822 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:29:01.940823 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:29:01.941046 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:29:01.941103 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:01.941106 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:01.942004 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:01.942005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:01.942524 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:01.942525 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:01.942955 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:01.942956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:01.942995 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:01.943483 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:29:01.943496 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:29:01.945409 (+ 1913us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.225.190:40039: Server oryx/127.22.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":64,"threads_started":1}
W20260928 06:29:01.945642 32592 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:29:01.940421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:35415 (local address 127.22.225.190:40039)
0928 06:29:01.940559 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 06:29:01.940561 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:29:01.940608 (+ 47us) server_negotiation.cc:407] Connection header received
0928 06:29:01.940635 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:29:01.940636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:29:01.940666 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:29:01.940725 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:29:01.941192 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:01.941915 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:01.942599 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:01.942885 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:01.942951 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:01.945517 (+ 2566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.225.129:35415: BlockingRecv error: recv got EOF from 127.22.225.129:35415 (error 108)
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":64,"threads_started":1}
Sep 28 06:29:02 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.130@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:29:02 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.130@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:29:02.069425 32593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:02.064132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:40039 (local address 127.22.225.130:41475)
0928 06:29:02.064260 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 06:29:02.064272 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:29:02.064349 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:29:02.064503 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:29:02.064503 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:29:02.064752 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:29:02.064845 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:02.064848 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:02.065521 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:02.065523 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:02.066499 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:02.066502 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:02.066879 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:02.066880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:02.066940 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:02.067378 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:29:02.067383 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:29:02.069329 (+ 1946us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.225.190:40039: Server oryx/127.22.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":70,"threads_started":1}
W20260928 06:29:02.069474 32592 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:29:02.064218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:41475 (local address 127.22.225.190:40039)
0928 06:29:02.064269 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 06:29:02.064271 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:29:02.064359 (+ 88us) server_negotiation.cc:407] Connection header received
0928 06:29:02.064413 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:29:02.064414 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:29:02.064429 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:29:02.064502 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:29:02.064933 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:02.065429 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:02.066591 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:02.066803 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:02.066857 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:02.069411 (+ 2554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.225.130:41475: BlockingRecv error: recv got EOF from 127.22.225.130:41475 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 28 06:29:02 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.131@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:29:02 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.131@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:29:02.206560 32594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:02.201671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:40039 (local address 127.22.225.131:38747)
0928 06:29:02.201865 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 06:29:02.201875 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:29:02.201957 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:29:02.202109 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:29:02.202110 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:29:02.202290 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:29:02.202335 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:02.202337 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:02.203016 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:02.203018 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:02.203554 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:02.203556 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:02.203920 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:02.203921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:02.203977 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:02.204493 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:29:02.204498 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:29:02.206460 (+ 1962us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.225.190:40039: Server oryx/127.22.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":95,"threads_started":1}
W20260928 06:29:02.206604 32592 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:29:02.201740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:38747 (local address 127.22.225.190:40039)
0928 06:29:02.201818 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 06:29:02.201820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:29:02.201951 (+ 131us) server_negotiation.cc:407] Connection header received
0928 06:29:02.202015 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:29:02.202016 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:29:02.202030 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:29:02.202104 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:29:02.202418 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:02.202915 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:02.203639 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:02.203834 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:02.203919 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:02.206539 (+ 2620us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.225.131:38747: BlockingRecv error: recv got EOF from 127.22.225.131:38747 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 28 06:29:02 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.129@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:29:02 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.129@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:29:02.951056 32595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:02.945965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:40039 (local address 127.22.225.129:34369)
0928 06:29:02.946121 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 06:29:02.946134 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:29:02.946184 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:29:02.946349 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:29:02.946350 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:29:02.946592 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:29:02.946649 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:02.946651 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:02.947636 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:02.947637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:02.948148 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:02.948149 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:02.948673 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:02.948674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:02.948720 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:02.949204 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:29:02.949209 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:29:02.950989 (+ 1780us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.225.190:40039: Server oryx/127.22.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":68,"threads_started":1}
W20260928 06:29:02.951265 32596 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:29:02.946030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:34369 (local address 127.22.225.190:40039)
0928 06:29:02.946153 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 06:29:02.946159 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:29:02.946171 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:29:02.946244 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:29:02.946244 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:29:02.946254 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:29:02.946318 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:29:02.946746 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:02.947543 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:02.948222 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:02.948541 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:02.948644 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:02.951124 (+ 2480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.225.129:34369: BlockingRecv error: recv got EOF from 127.22.225.129:34369 (error 108)
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":68,"threads_started":1}
Sep 28 06:29:03 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.130@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:29:03 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.130@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:29:03.074360 32597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:03.069910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:40039 (local address 127.22.225.130:57129)
0928 06:29:03.070033 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 06:29:03.070045 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:29:03.070105 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:29:03.070252 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:29:03.070253 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:29:03.070424 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:29:03.070468 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:03.070470 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:03.071155 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:03.071156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:03.071669 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:03.071671 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:03.072033 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:03.072033 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:03.072090 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:03.072552 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:29:03.072564 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:29:03.074268 (+ 1704us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.225.190:40039: Server oryx/127.22.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":59,"threads_started":1}
W20260928 06:29:03.074409 32596 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:29:03.070035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:57129 (local address 127.22.225.190:40039)
0928 06:29:03.070103 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 06:29:03.070105 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:29:03.070115 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:29:03.070175 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:29:03.070176 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:29:03.070189 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:29:03.070243 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:29:03.070553 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:03.071065 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:03.071752 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:03.071944 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:03.072031 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:03.074346 (+ 2315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.225.130:57129: BlockingRecv error: recv got EOF from 127.22.225.130:57129 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 28 06:29:03 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.131@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:29:03 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.131@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:29:03.212255 32598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:03.207051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:40039 (local address 127.22.225.131:40801)
0928 06:29:03.207210 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 06:29:03.207222 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:29:03.207298 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:29:03.207442 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:29:03.207443 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:29:03.207636 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:29:03.207691 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:03.207693 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:03.208346 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:03.208348 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:03.209203 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:03.209205 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:03.209600 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:03.209601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:03.209669 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:03.210180 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:29:03.210185 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:29:03.212156 (+ 1971us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.225.190:40039: Server oryx/127.22.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":80,"threads_started":1}
W20260928 06:29:03.212301 32596 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:29:03.207154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:40801 (local address 127.22.225.190:40039)
0928 06:29:03.207206 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 06:29:03.207208 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:29:03.207277 (+ 69us) server_negotiation.cc:407] Connection header received
0928 06:29:03.207356 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:29:03.207357 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:29:03.207369 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:29:03.207417 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:29:03.207769 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:03.208263 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:03.209279 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:03.209531 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:03.209586 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:03.212237 (+ 2651us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.225.131:40801: BlockingRecv error: recv got EOF from 127.22.225.131:40801 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 28 06:29:03 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.129@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:29:03 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.129@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:29:03.963747 32599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:03.951550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:40039 (local address 127.22.225.129:52127)
0928 06:29:03.951682 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 06:29:03.951695 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:29:03.951746 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:29:03.951894 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:29:03.951895 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:29:03.952048 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:29:03.952095 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:03.952097 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:03.953054 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:03.953055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:03.953558 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:03.953560 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:03.954014 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:03.954014 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:03.954052 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:03.961749 (+ 7697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:29:03.961753 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:29:03.963687 (+ 1934us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.225.190:40039: Server oryx/127.22.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":82,"threads_started":1}
W20260928 06:29:03.963871 32600 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:29:03.951611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:52127 (local address 127.22.225.190:40039)
0928 06:29:03.951726 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 06:29:03.951728 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:29:03.951739 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:29:03.951791 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:29:03.951791 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:29:03.951802 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:29:03.951884 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:29:03.952170 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:03.952978 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:03.953645 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:03.953938 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:03.954010 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:03.963801 (+ 9791us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.225.129:52127: BlockingRecv error: recv got EOF from 127.22.225.129:52127 (error 108)
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":61,"threads_started":1}
Sep 28 06:29:04 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.130@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:29:04 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.130@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:29:04.079741 32601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:04.074875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:40039 (local address 127.22.225.130:42025)
0928 06:29:04.074982 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 06:29:04.075002 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 06:29:04.075090 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:29:04.075262 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:29:04.075263 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:29:04.075431 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:29:04.075476 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:04.075478 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:04.076171 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:04.076172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:04.076762 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:04.076765 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:04.077140 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:04.077141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:04.077196 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:04.077668 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:29:04.077674 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:29:04.079667 (+ 1993us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.225.190:40039: Server oryx/127.22.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":55,"threads_started":1}
W20260928 06:29:04.079880 32600 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:29:04.074925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:42025 (local address 127.22.225.190:40039)
0928 06:29:04.074975 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 06:29:04.074977 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:29:04.075069 (+ 92us) server_negotiation.cc:407] Connection header received
0928 06:29:04.075174 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:29:04.075174 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:29:04.075186 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:29:04.075258 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:29:04.075564 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:04.076092 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:04.076858 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:04.077056 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:04.077139 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:04.079810 (+ 2671us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.225.130:42025: BlockingRecv error: recv got EOF from 127.22.225.130:42025 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 28 06:29:04 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.131@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:29:04 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.131@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:29:04.217510 32602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:04.212721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:40039 (local address 127.22.225.131:39983)
0928 06:29:04.212850 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 06:29:04.212861 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:29:04.212946 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:29:04.213095 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:29:04.213096 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:29:04.213268 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:29:04.213314 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:04.213316 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:04.214075 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:04.214076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:04.214636 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:04.214637 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:04.215035 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:04.215036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:04.215123 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:04.215607 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:29:04.215612 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:29:04.217406 (+ 1794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.225.190:40039: Server oryx/127.22.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":66,"threads_started":1}
W20260928 06:29:04.217545 32600 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:29:04.212772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:39983 (local address 127.22.225.190:40039)
0928 06:29:04.212819 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 06:29:04.212821 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:29:04.212928 (+ 107us) server_negotiation.cc:407] Connection header received
0928 06:29:04.213015 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:29:04.213015 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:29:04.213029 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:29:04.213084 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:29:04.213389 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:04.213998 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:04.214736 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:04.214944 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:04.215035 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:04.217485 (+ 2450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.225.131:39983: BlockingRecv error: recv got EOF from 127.22.225.131:39983 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 28 06:29:04 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.129@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:29:04 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.129@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:29:04.969674 32603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:04.964391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:40039 (local address 127.22.225.129:33263)
0928 06:29:04.964536 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 06:29:04.964549 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:29:04.964641 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:29:04.964826 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:29:04.964827 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:29:04.965003 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:29:04.965050 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:04.965052 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:04.965972 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:04.965973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:04.966499 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:04.966501 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:04.966940 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:04.966941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:04.966984 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:04.967520 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:29:04.967524 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:29:04.969583 (+ 2059us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.225.190:40039: Server oryx/127.22.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":67,"threads_started":1}
W20260928 06:29:04.969758 32604 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:29:04.964479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:33263 (local address 127.22.225.190:40039)
0928 06:29:04.964659 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 06:29:04.964661 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:29:04.964671 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:29:04.964698 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:29:04.964699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:29:04.964712 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:29:04.964783 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:29:04.965127 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:04.965882 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:04.966585 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:04.966864 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:04.966935 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:04.969687 (+ 2752us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.225.129:33263: BlockingRecv error: recv got EOF from 127.22.225.129:33263 (error 108)
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":70,"threads_started":1}
Sep 28 06:29:05 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.130@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:29:05 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.130@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:29:05.085275 32605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:05.080339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:40039 (local address 127.22.225.130:42451)
0928 06:29:05.080464 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 06:29:05.080480 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 06:29:05.080537 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:29:05.080709 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:29:05.080711 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:29:05.080870 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:29:05.080918 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:05.080920 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:05.081767 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:05.081768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:05.082297 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:05.082298 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:05.082723 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:05.082724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:05.082764 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:05.083295 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:29:05.083299 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:29:05.085205 (+ 1906us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.225.190:40039: Server oryx/127.22.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":69,"threads_started":1}
W20260928 06:29:05.085400 32604 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:29:05.080410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:42451 (local address 127.22.225.190:40039)
0928 06:29:05.080465 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 06:29:05.080467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:29:05.080525 (+ 58us) server_negotiation.cc:407] Connection header received
0928 06:29:05.080634 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:29:05.080635 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:29:05.080647 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:29:05.080697 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:29:05.080986 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:05.081686 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:05.082376 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:05.082646 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:05.082719 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:05.085337 (+ 2618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.225.130:42451: BlockingRecv error: recv got EOF from 127.22.225.130:42451 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 28 06:29:05 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.131@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:29:05 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.131@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:29:05.223203 32606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:05.218157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:40039 (local address 127.22.225.131:59915)
0928 06:29:05.218288 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 06:29:05.218303 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:29:05.218385 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:29:05.218595 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:29:05.218597 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:29:05.218861 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:29:05.218921 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:05.218923 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:05.219824 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:05.219825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:05.220342 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:05.220343 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:05.220856 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:05.220856 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:05.220899 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:05.221415 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:29:05.221420 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:29:05.223146 (+ 1726us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.225.190:40039: Server oryx/127.22.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":62,"threads_started":1}
W20260928 06:29:05.223387 32604 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:29:05.218237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:59915 (local address 127.22.225.190:40039)
0928 06:29:05.218292 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 06:29:05.218294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:29:05.218377 (+ 83us) server_negotiation.cc:407] Connection header received
0928 06:29:05.218461 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:29:05.218462 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:29:05.218483 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:29:05.218560 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:29:05.218987 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:05.219748 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:05.220428 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:05.220781 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:05.220858 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:05.223257 (+ 2399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.225.131:59915: BlockingRecv error: recv got EOF from 127.22.225.131:59915 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 28 06:29:05 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.129@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:29:05 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.129@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:29:05.975415 32607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:05.970242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:40039 (local address 127.22.225.129:44001)
0928 06:29:05.970394 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 06:29:05.970407 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:29:05.970473 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:29:05.970625 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:29:05.970626 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:29:05.970781 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:29:05.970829 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:05.970831 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:05.971786 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:05.971786 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:05.972279 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:05.972281 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:05.972810 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:05.972811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:05.972855 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:05.973404 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:29:05.973409 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:29:05.975358 (+ 1949us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.225.190:40039: Server oryx/127.22.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":70,"threads_started":1}
W20260928 06:29:05.975562 32608 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:29:05.970347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:44001 (local address 127.22.225.190:40039)
0928 06:29:05.970487 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 06:29:05.970489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:29:05.970503 (+ 14us) server_negotiation.cc:407] Connection header received
0928 06:29:05.970529 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:29:05.970530 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:29:05.970542 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:29:05.970615 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:29:05.970901 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:05.971701 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:05.972365 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:05.972715 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:05.972789 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:05.975462 (+ 2673us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.225.129:44001: BlockingRecv error: recv got EOF from 127.22.225.129:44001 (error 108)
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":65,"threads_started":1}
Sep 28 06:29:06 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.130@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:29:06 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.130@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:29:06.091532 32609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:06.085911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:40039 (local address 127.22.225.130:40925)
0928 06:29:06.086045 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 06:29:06.086056 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:29:06.086123 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:29:06.086264 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:29:06.086265 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:29:06.086418 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:29:06.086466 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:06.086468 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:06.087791 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:06.087791 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:06.088286 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:06.088287 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:06.088820 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:06.088821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:06.088863 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:06.089394 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:29:06.089399 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:29:06.091466 (+ 2067us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.225.190:40039: Server oryx/127.22.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":91,"threads_started":1}
W20260928 06:29:06.091668 32608 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:29:06.085981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:40925 (local address 127.22.225.190:40039)
0928 06:29:06.086037 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 06:29:06.086039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:29:06.086130 (+ 91us) server_negotiation.cc:407] Connection header received
0928 06:29:06.086168 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:29:06.086169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:29:06.086183 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:29:06.086249 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:29:06.086546 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:06.087712 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:06.088372 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:06.088744 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:06.088821 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:06.091593 (+ 2772us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.225.130:40925: BlockingRecv error: recv got EOF from 127.22.225.130:40925 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Sep 28 06:29:06 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.131@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:29:06 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.131@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:29:06.228976 32610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:06.223686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:40039 (local address 127.22.225.131:41161)
0928 06:29:06.223832 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 06:29:06.223844 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:29:06.223898 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:29:06.224202 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:29:06.224203 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:29:06.224382 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:29:06.224447 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:06.224449 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:06.225300 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:06.225301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:06.225847 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:06.225849 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:06.226260 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:06.226261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:06.226299 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:06.226798 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:29:06.226802 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:29:06.228884 (+ 2082us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.225.190:40039: Server oryx/127.22.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":95,"threads_started":1}
W20260928 06:29:06.229041 32608 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:29:06.223881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:41161 (local address 127.22.225.190:40039)
0928 06:29:06.223946 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 06:29:06.223948 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:29:06.223959 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:29:06.224061 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:29:06.224061 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:29:06.224077 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:29:06.224150 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:29:06.224523 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:06.225217 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:06.225930 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:06.226189 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:06.226256 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:06.228972 (+ 2716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.225.131:41161: BlockingRecv error: recv got EOF from 127.22.225.131:41161 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 28 06:29:06 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.129@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:29:06 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.129@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:29:06.981154 32611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:06.975942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:40039 (local address 127.22.225.129:57813)
0928 06:29:06.976074 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 06:29:06.976092 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 06:29:06.976187 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:29:06.976341 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:29:06.976342 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:29:06.976529 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:29:06.976640 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:06.976642 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:06.977332 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:06.977334 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:06.977838 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:06.977839 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:06.978205 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:06.978206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:06.978264 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:06.978801 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:29:06.978807 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:29:06.981050 (+ 2243us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.225.190:40039: Server oryx/127.22.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":72,"threads_started":1}
W20260928 06:29:06.981192 32612 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:29:06.975980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:57813 (local address 127.22.225.190:40039)
0928 06:29:06.976109 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 06:29:06.976111 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:29:06.976167 (+ 56us) server_negotiation.cc:407] Connection header received
0928 06:29:06.976261 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:29:06.976262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:29:06.976276 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:29:06.976337 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:29:06.976724 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:06.977240 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:06.977917 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:06.978131 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:06.978206 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:06.981140 (+ 2934us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.225.129:57813: BlockingRecv error: recv got EOF from 127.22.225.129:57813 (error 108)
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":66,"threads_started":1}
Sep 28 06:29:07 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.130@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:29:07 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.130@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:29:07.097136 32613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:07.092080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:40039 (local address 127.22.225.130:44627)
0928 06:29:07.092192 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 06:29:07.092205 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:29:07.092287 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:29:07.092496 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:29:07.092497 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:29:07.092786 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:29:07.092846 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:07.092848 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:07.093516 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:07.093517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:07.094100 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:07.094102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:07.094478 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:07.094479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:07.094525 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:07.095070 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:29:07.095075 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:29:07.097022 (+ 1947us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.225.190:40039: Server oryx/127.22.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":67,"threads_started":1}
W20260928 06:29:07.097173 32612 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:29:07.092149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:44627 (local address 127.22.225.190:40039)
0928 06:29:07.092215 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 06:29:07.092217 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:29:07.092269 (+ 52us) server_negotiation.cc:407] Connection header received
0928 06:29:07.092365 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:29:07.092366 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:29:07.092379 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:29:07.092438 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:29:07.092945 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:07.093439 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:07.094177 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:07.094408 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:07.094464 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:07.097107 (+ 2643us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.225.130:44627: BlockingRecv error: recv got EOF from 127.22.225.130:44627 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 28 06:29:07 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.131@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:29:07 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.131@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:29:07.235508 32614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:07.229487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:40039 (local address 127.22.225.131:43531)
0928 06:29:07.229618 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 06:29:07.229630 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:29:07.229699 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:29:07.229853 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:29:07.229854 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:29:07.230077 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:29:07.230155 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:07.230157 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:07.231149 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:07.231150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:07.232052 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:07.232055 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:07.232534 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:07.232535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:07.232601 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:07.233121 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:29:07.233126 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:29:07.235394 (+ 2268us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.225.190:40039: Server oryx/127.22.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":66,"threads_started":1}
W20260928 06:29:07.235587 32612 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:29:07.229562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:43531 (local address 127.22.225.190:40039)
0928 06:29:07.229614 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 06:29:07.229616 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:29:07.229698 (+ 82us) server_negotiation.cc:407] Connection header received
0928 06:29:07.229775 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:29:07.229776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:29:07.229791 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:29:07.229838 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:29:07.230253 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:07.231068 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:07.232143 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:07.232446 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:07.232514 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:07.235515 (+ 3001us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.225.131:43531: BlockingRecv error: recv got EOF from 127.22.225.131:43531 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 28 06:29:07 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.129@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:29:07 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.129@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:29:07.986850 32615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:07.981690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:40039 (local address 127.22.225.129:45097)
0928 06:29:07.981817 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 06:29:07.981827 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:29:07.981873 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:29:07.982043 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:29:07.982044 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:29:07.982196 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:29:07.982245 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:07.982246 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:07.983214 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:07.983215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:07.983730 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:07.983732 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:07.984239 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:07.984239 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:07.984280 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:07.984809 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:29:07.984814 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:29:07.986790 (+ 1976us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.225.190:40039: Server oryx/127.22.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":68,"threads_started":1}
W20260928 06:29:07.987005 32616 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:29:07.981775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:45097 (local address 127.22.225.190:40039)
0928 06:29:07.981907 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 06:29:07.981909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:29:07.981918 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:29:07.981942 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:29:07.981943 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:29:07.981955 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:29:07.982039 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:29:07.982317 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:07.983125 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:07.983812 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:07.984151 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:07.984227 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:07.986922 (+ 2695us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.225.129:45097: BlockingRecv error: recv got EOF from 127.22.225.129:45097 (error 108)
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":77,"threads_started":1}
Sep 28 06:29:08 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.130@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:29:08 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.130@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:29:08.102835 32617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:08.097700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:40039 (local address 127.22.225.130:44909)
0928 06:29:08.097876 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 06:29:08.097887 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:29:08.097962 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:29:08.098110 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:29:08.098111 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:29:08.098273 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:29:08.098321 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:08.098323 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:08.099168 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:08.099168 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:08.099988 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:08.099992 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:08.100393 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:08.100394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:08.100437 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:08.100948 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:29:08.100953 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:29:08.102769 (+ 1816us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.225.190:40039: Server oryx/127.22.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":92,"threads_started":1}
W20260928 06:29:08.103051 32616 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:29:08.097775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:44909 (local address 127.22.225.190:40039)
0928 06:29:08.097860 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0928 06:29:08.097863 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:29:08.097963 (+ 100us) server_negotiation.cc:407] Connection header received
0928 06:29:08.098017 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:29:08.098018 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:29:08.098031 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:29:08.098094 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:29:08.098397 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:08.099082 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:08.100054 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:08.100317 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:08.100387 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:08.102901 (+ 2514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.225.130:44909: BlockingRecv error: recv got EOF from 127.22.225.130:44909 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 28 06:29:08 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.131@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:29:08 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.131@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:29:08.241475 32618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:08.236210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:40039 (local address 127.22.225.131:41011)
0928 06:29:08.236421 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 06:29:08.236442 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 06:29:08.236509 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:29:08.236675 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:29:08.236675 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:29:08.236843 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:29:08.236889 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:08.236891 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:08.237774 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:08.237775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:08.238312 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:08.238313 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:08.238761 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:08.238761 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:08.238799 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:08.239311 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:29:08.239315 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:29:08.241404 (+ 2089us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.225.190:40039: Server oryx/127.22.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":89,"threads_started":1}
W20260928 06:29:08.241581 32616 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:29:08.236371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:41011 (local address 127.22.225.190:40039)
0928 06:29:08.236526 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 06:29:08.236528 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:29:08.236542 (+ 14us) server_negotiation.cc:407] Connection header received
0928 06:29:08.236569 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:29:08.236570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:29:08.236606 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:29:08.236670 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:29:08.236961 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:08.237699 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:08.238398 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:08.238686 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:08.238757 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:08.241519 (+ 2762us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.225.131:41011: BlockingRecv error: recv got EOF from 127.22.225.131:41011 (error 108)
Metrics: {"server-negotiator.queue_time_us":127}
Sep 28 06:29:08 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.129@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:29:08 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.129@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:29:08.998575 32619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:08.987705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:40039 (local address 127.22.225.129:35887)
0928 06:29:08.987861 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 06:29:08.987871 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:29:08.987939 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:29:08.988124 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:29:08.988125 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:29:08.988295 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:29:08.988341 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:08.988343 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:08.989258 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:08.989259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:08.989812 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:08.989814 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:08.990278 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:08.990279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:08.990340 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:08.996433 (+ 6093us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:29:08.996439 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:29:08.998489 (+ 2050us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.225.190:40039: Server oryx/127.22.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":93,"threads_started":1}
W20260928 06:29:08.998713 32620 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:29:08.987795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:35887 (local address 127.22.225.190:40039)
0928 06:29:08.987922 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 06:29:08.987924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:29:08.987938 (+ 14us) server_negotiation.cc:407] Connection header received
0928 06:29:08.988009 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:29:08.988010 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:29:08.988024 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:29:08.988111 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:29:08.988424 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:08.989165 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:08.989891 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:08.990153 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:08.990230 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:08.998638 (+ 8408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.225.129:35887: BlockingRecv error: recv got EOF from 127.22.225.129:35887 (error 108)
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":66,"threads_started":1}
Sep 28 06:29:09 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.130@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:29:09 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.130@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:29:09.108319 32621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:09.103450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:40039 (local address 127.22.225.130:36313)
0928 06:29:09.103573 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 06:29:09.103587 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:29:09.103660 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:29:09.103825 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:29:09.103826 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:29:09.104026 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:29:09.104071 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:09.104073 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:09.104852 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:09.104853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:09.105417 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:09.105419 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:09.105813 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:09.105814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:09.105871 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:09.106433 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:29:09.106437 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:29:09.108206 (+ 1769us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.225.190:40039: Server oryx/127.22.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":69,"threads_started":1}
W20260928 06:29:09.108364 32620 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:29:09.103548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:36313 (local address 127.22.225.190:40039)
0928 06:29:09.103610 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 06:29:09.103612 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:29:09.103642 (+ 30us) server_negotiation.cc:407] Connection header received
0928 06:29:09.103742 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:29:09.103743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:29:09.103755 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:29:09.103810 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:29:09.104162 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:09.104760 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:09.105513 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:09.105733 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:09.105816 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:09.108298 (+ 2482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.225.130:36313: BlockingRecv error: recv got EOF from 127.22.225.130:36313 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 28 06:29:09 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.131@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:29:09 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.131@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:29:09.247617 32622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:09.242164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:40039 (local address 127.22.225.131:55099)
0928 06:29:09.242292 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 06:29:09.242303 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:29:09.242366 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:29:09.242511 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:29:09.242511 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:29:09.242671 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:29:09.242714 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:09.242715 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:09.243426 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:09.243428 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:09.244052 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:09.244055 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:09.244475 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:09.244476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:09.244538 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:09.245204 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:29:09.245210 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:29:09.247496 (+ 2286us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.225.190:40039: Server oryx/127.22.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":74,"threads_started":1}
W20260928 06:29:09.247640 32620 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:29:09.242239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:55099 (local address 127.22.225.190:40039)
0928 06:29:09.242294 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 06:29:09.242296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:29:09.242373 (+ 77us) server_negotiation.cc:407] Connection header received
0928 06:29:09.242406 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:29:09.242407 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:29:09.242423 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:29:09.242500 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:29:09.242826 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:09.243336 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:09.244169 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:09.244387 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:09.244473 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:09.247593 (+ 3120us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.225.131:55099: BlockingRecv error: recv got EOF from 127.22.225.131:55099 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 28 06:29:09 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.129@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:29:09 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.129@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:29:10.004783 32623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:09.999256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:40039 (local address 127.22.225.129:46303)
0928 06:29:09.999413 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 06:29:09.999431 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 06:29:09.999484 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:29:09.999718 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:29:09.999719 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:29:09.999907 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:29:09.999956 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:09.999957 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:10.001000 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:10.001001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:10.001517 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:10.001518 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:10.002024 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:10.002025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:10.002065 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:10.002590 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:29:10.002595 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:29:10.004723 (+ 2128us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.225.190:40039: Server oryx/127.22.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":73,"threads_started":1}
W20260928 06:29:10.004922 32624 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:29:09.999365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:46303 (local address 127.22.225.190:40039)
0928 06:29:09.999500 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 06:29:09.999503 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:29:09.999517 (+ 14us) server_negotiation.cc:407] Connection header received
0928 06:29:09.999550 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:29:09.999551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:29:09.999588 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:29:09.999680 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:29:10.000049 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:10.000911 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:10.001608 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:10.001923 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:10.002000 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:10.004846 (+ 2846us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.225.129:46303: BlockingRecv error: recv got EOF from 127.22.225.129:46303 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":67,"threads_started":1}
Sep 28 06:29:10 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.130@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:29:10 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.130@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:29:10.114272 32625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:10.108983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:40039 (local address 127.22.225.130:48361)
0928 06:29:10.109104 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 06:29:10.109118 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:29:10.109187 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:29:10.109317 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:29:10.109318 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:29:10.109506 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:29:10.109550 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:10.109553 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:10.110364 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:10.110365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:10.110919 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:10.110920 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:10.111332 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:10.111334 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:10.111398 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:10.111938 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:29:10.111944 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:29:10.114164 (+ 2220us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.225.190:40039: Server oryx/127.22.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":69,"threads_started":1}
W20260928 06:29:10.114310 32624 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:29:10.109088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:48361 (local address 127.22.225.190:40039)
0928 06:29:10.109179 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0928 06:29:10.109181 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:29:10.109193 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:29:10.109249 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:29:10.109250 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:29:10.109263 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:29:10.109316 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:29:10.109643 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:10.110280 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:10.111027 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:10.111241 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:10.111329 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:10.114244 (+ 2915us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.225.130:48361: BlockingRecv error: recv got EOF from 127.22.225.130:48361 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 28 06:29:10 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.131@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:29:10 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.131@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:29:10.253593 32626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:10.248320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:40039 (local address 127.22.225.131:41675)
0928 06:29:10.248501 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 06:29:10.248512 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:29:10.248566 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:29:10.248770 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:29:10.248771 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:29:10.248940 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:29:10.248986 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:10.248987 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:10.249928 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:10.249929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:10.250442 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:10.250443 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:10.250922 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:10.250924 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:10.250971 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:10.251480 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:29:10.251485 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:29:10.253497 (+ 2012us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.225.190:40039: Server oryx/127.22.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":103,"threads_started":1}
W20260928 06:29:10.253636 32624 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:29:10.248421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:41675 (local address 127.22.225.190:40039)
0928 06:29:10.248497 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 06:29:10.248499 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:29:10.248561 (+ 62us) server_negotiation.cc:407] Connection header received
0928 06:29:10.248663 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:29:10.248664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:29:10.248680 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:29:10.248759 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:29:10.249061 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:10.249842 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:10.250526 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:10.250836 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:10.250908 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:10.253571 (+ 2663us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.225.131:41675: BlockingRecv error: recv got EOF from 127.22.225.131:41675 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 28 06:29:11 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.129@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:29:11 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.129@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:29:11.011091 32627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:11.005440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:40039 (local address 127.22.225.129:56919)
0928 06:29:11.005598 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 06:29:11.005609 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:29:11.005657 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:29:11.005838 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:29:11.005839 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:29:11.006019 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:29:11.006068 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:11.006070 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:11.007045 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:11.007046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:11.007593 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:11.007596 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:11.008047 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:11.008047 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:11.008087 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:11.008665 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:29:11.008670 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:29:11.011007 (+ 2337us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.225.190:40039: Server oryx/127.22.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":94,"threads_started":1}
W20260928 06:29:11.011150 32628 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:29:11.005514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:56919 (local address 127.22.225.190:40039)
0928 06:29:11.005636 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 06:29:11.005638 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:29:11.005652 (+ 14us) server_negotiation.cc:407] Connection header received
0928 06:29:11.005708 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:29:11.005709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:29:11.005724 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:29:11.005818 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:29:11.006151 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:11.006941 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:11.007687 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:11.007965 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:11.008049 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:11.011096 (+ 3047us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.225.129:56919: BlockingRecv error: recv got EOF from 127.22.225.129:56919 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":62,"threads_started":1}
Sep 28 06:29:11 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.130@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:29:11 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.130@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:29:11.119881 32629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:11.114906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:40039 (local address 127.22.225.130:53587)
0928 06:29:11.115050 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 06:29:11.115063 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:29:11.115156 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:29:11.115309 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:29:11.115310 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:29:11.115499 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:29:11.115544 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:11.115546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:11.116223 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:11.116224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:11.116815 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:11.116817 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:11.117199 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:11.117200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:11.117262 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:11.117758 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:29:11.117762 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:29:11.119786 (+ 2024us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.225.190:40039: Server oryx/127.22.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":61,"threads_started":1}
W20260928 06:29:11.119928 32628 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:29:11.115005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:53587 (local address 127.22.225.190:40039)
0928 06:29:11.115053 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 06:29:11.115055 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:29:11.115138 (+ 83us) server_negotiation.cc:407] Connection header received
0928 06:29:11.115236 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:29:11.115237 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:29:11.115251 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:29:11.115306 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:29:11.115631 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:11.116137 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:11.116898 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:11.117113 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:11.117199 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:11.119864 (+ 2665us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.225.130:53587: BlockingRecv error: recv got EOF from 127.22.225.130:53587 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 28 06:29:11 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.131@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:29:11 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.131@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:29:11.259172 32630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:11.254174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:40039 (local address 127.22.225.131:37343)
0928 06:29:11.254310 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 06:29:11.254322 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:29:11.254395 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:29:11.254558 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:29:11.254559 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:29:11.254745 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:29:11.254787 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:11.254789 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:11.255475 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:11.255476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:11.256034 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:11.256036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:11.256402 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:11.256403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:11.256459 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:11.257002 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:29:11.257007 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:29:11.259048 (+ 2041us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.225.190:40039: Server oryx/127.22.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":84,"threads_started":1}
W20260928 06:29:11.259204 32628 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:29:11.254252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:37343 (local address 127.22.225.190:40039)
0928 06:29:11.254322 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 06:29:11.254324 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:29:11.254376 (+ 52us) server_negotiation.cc:407] Connection header received
0928 06:29:11.254458 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:29:11.254460 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:29:11.254473 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:29:11.254549 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:29:11.254874 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:11.255379 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:11.256123 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:11.256323 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:11.256404 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:11.259129 (+ 2725us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.225.131:37343: BlockingRecv error: recv got EOF from 127.22.225.131:37343 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 28 06:29:12 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.129@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:29:12 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.129@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:29:12.016515 32631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:12.011604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:40039 (local address 127.22.225.129:39035)
0928 06:29:12.011727 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 06:29:12.011745 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 06:29:12.011802 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:29:12.011969 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:29:12.011969 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:29:12.012140 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:29:12.012186 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:12.012188 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:12.013104 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:12.013104 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:12.013640 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:12.013641 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:12.014072 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:12.014073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:12.014138 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:12.014664 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:29:12.014669 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:29:12.016441 (+ 1772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.225.190:40039: Server oryx/127.22.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":70,"threads_started":1}
W20260928 06:29:12.016645 32632 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:29:12.011674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:39035 (local address 127.22.225.190:40039)
0928 06:29:12.011795 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 06:29:12.011797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:29:12.011808 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:29:12.011861 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:29:12.011861 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:29:12.011874 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:29:12.011958 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:29:12.012267 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:12.013020 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:12.013719 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:12.014000 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:12.014072 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:12.016556 (+ 2484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.225.129:39035: BlockingRecv error: recv got EOF from 127.22.225.129:39035 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":65,"threads_started":1}
Sep 28 06:29:12 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.130@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:29:12 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.130@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:29:12.126045 32633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:12.120778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:40039 (local address 127.22.225.130:51495)
0928 06:29:12.120949 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 06:29:12.120962 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:29:12.121034 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:29:12.121176 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:29:12.121177 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:29:12.121380 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:29:12.121435 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:12.121438 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:12.122077 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:12.122078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:12.122804 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:12.122807 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:12.123133 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:12.123134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:12.123186 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:12.123645 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:29:12.123650 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:29:12.125970 (+ 2320us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.225.190:40039: Server oryx/127.22.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":90,"threads_started":1}
W20260928 06:29:12.126155 32632 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:29:12.120842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:51495 (local address 127.22.225.190:40039)
0928 06:29:12.120916 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 06:29:12.120918 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:29:12.121041 (+ 123us) server_negotiation.cc:407] Connection header received
0928 06:29:12.121097 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:29:12.121098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:29:12.121110 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:29:12.121153 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:29:12.121508 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:12.122001 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:12.122884 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:12.123066 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:12.123113 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:12.126070 (+ 2957us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.225.130:51495: BlockingRecv error: recv got EOF from 127.22.225.130:51495 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 28 06:29:12 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.131@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:29:12 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.131@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:29:12.265071 32634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:12.259785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:40039 (local address 127.22.225.131:55837)
0928 06:29:12.259964 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 06:29:12.259975 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:29:12.260040 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:29:12.260172 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:29:12.260172 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:29:12.260344 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:29:12.260388 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:12.260390 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:12.261325 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:12.261326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:12.261875 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:12.261877 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:12.262363 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:12.262364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:12.262403 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:12.262915 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:29:12.262927 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:29:12.264933 (+ 2006us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.225.190:40039: Server oryx/127.22.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":80,"threads_started":1}
W20260928 06:29:12.265143 32632 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:29:12.259878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:55837 (local address 127.22.225.190:40039)
0928 06:29:12.259926 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 06:29:12.259928 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:29:12.260053 (+ 125us) server_negotiation.cc:407] Connection header received
0928 06:29:12.260083 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:29:12.260084 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:29:12.260100 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:29:12.260159 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:29:12.260462 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:12.261236 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:12.261973 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:12.262287 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:12.262359 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:12.265083 (+ 2724us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.225.131:55837: BlockingRecv error: recv got EOF from 127.22.225.131:55837 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 28 06:29:13 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.129@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:29:13 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.129@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:29:13.022378 32635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:13.017200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:40039 (local address 127.22.225.129:45785)
0928 06:29:13.017365 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 06:29:13.017377 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:29:13.017448 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:29:13.017627 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:29:13.017628 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:29:13.017782 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:29:13.017831 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:13.017833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:13.018697 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:13.018698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:13.019223 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:13.019225 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:13.019671 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:13.019672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:13.019710 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:13.020206 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:29:13.020212 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:29:13.022318 (+ 2106us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.225.190:40039: Server oryx/127.22.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":93,"threads_started":1}
W20260928 06:29:13.022533 32636 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:29:13.017294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:45785 (local address 127.22.225.190:40039)
0928 06:29:13.017425 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 06:29:13.017429 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:29:13.017439 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:29:13.017524 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:29:13.017525 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:29:13.017539 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:29:13.017609 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:29:13.017900 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:13.018609 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:13.019312 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:13.019589 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:13.019668 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:13.022455 (+ 2787us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.225.129:45785: BlockingRecv error: recv got EOF from 127.22.225.129:45785 (error 108)
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":57,"threads_started":1}
Sep 28 06:29:13 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.130@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:29:13 dist-test-slave-d59m krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.225.130@KRBTEST.COM for oryx/127.22.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:29:13.131132 32637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:13.126547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:40039 (local address 127.22.225.130:51261)
0928 06:29:13.126704 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 06:29:13.126715 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:29:13.126798 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:29:13.126991 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:29:13.126992 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:29:13.127169 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:29:13.127216 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:13.127218 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:13.127886 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:13.127887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:13.128408 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:13.128410 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:13.128844 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:13.128845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:13.128900 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:13.129334 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:29:13.129339 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:29:13.131062 (+ 1723us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.225.190:40039: Server oryx/127.22.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":90,"threads_started":1}
W20260928 06:29:13.131250 32636 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:29:13.126624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:51261 (local address 127.22.225.190:40039)
0928 06:29:13.126691 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 06:29:13.126694 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:29:13.126779 (+ 85us) server_negotiation.cc:407] Connection header received
0928 06:29:13.126866 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:29:13.126867 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:29:13.126883 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:29:13.126982 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:29:13.127294 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:13.127792 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:13.128502 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:13.128741 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:13.128845 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:13.131154 (+ 2309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.225.130:51261: BlockingRecv error: recv got EOF from 127.22.225.130:51261 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 06:29:13.177304 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 32161
I20260928 06:29:13.181898 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 32313
I20260928 06:29:13.187248 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 32445
I20260928 06:29:13.191870 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 32091
2026-09-28T06:29:13Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17623 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T06:29:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:29:13Z Disabled control of system clock
I20260928 06:29:13.222402 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576899143097-23430-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576899143097-23430-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576899143097-23430-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:40563
--webserver_interface=127.22.225.190
--webserver_port=0
--builtin_ntp_servers=127.22.225.148:34683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.225.190:40563
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 06:29:13.338013 32641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:29:13.338306 32641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:29:13.338337 32641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:29:13.339798 32641 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:29:13.339854 32641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:29:13.339869 32641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:29:13.339903 32641 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:29:13.339929 32641 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:29:13.341679 32641 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:34683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576899143097-23430-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576899143097-23430-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.225.190:40563
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:40563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576899143097-23430-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:29:13.341953 32641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:29:13.342296 32641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:29:13.347756 32646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:29:13.347764 32649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:29:13.347821 32647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:29:13.348084 32641 server_base.cc:1034] running on GCE node
I20260928 06:29:13.348412 32641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:29:13.348802 32641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:29:13.349941 32641 hybrid_clock.cc:648] HybridClock initialized: now 1790576953349925 us; error 31 us; skew 500 ppm
I20260928 06:29:13.351229 32641 webserver.cc:460] Webserver started at http://127.22.225.190:46585/ using document root <none> and password file <none>
I20260928 06:29:13.351486 32641 fs_manager.cc:359] Metadata directory not provided
I20260928 06:29:13.351565 32641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:29:13.351720 32641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:29:13.352885 32641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576899143097-23430-0/minicluster-data/master-0/data/instance:
uuid: "101546e0c10f4093989716a4818eb766"
format_stamp: "Formatted at 2026-09-28 06:29:13 on dist-test-slave-d59m"
server_key: "02f2938c0cd3c0c27c4faa31c0a97993"
server_key_iv: "cca2997193d060011f0fec64fde565bc"
server_key_version: "encryptionkey@0"
I20260928 06:29:13.353252 32641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576899143097-23430-0/minicluster-data/master-0/wal/instance:
uuid: "101546e0c10f4093989716a4818eb766"
format_stamp: "Formatted at 2026-09-28 06:29:13 on dist-test-slave-d59m"
server_key: "02f2938c0cd3c0c27c4faa31c0a97993"
server_key_iv: "cca2997193d060011f0fec64fde565bc"
server_key_version: "encryptionkey@0"
I20260928 06:29:13.354604 32641 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:29:13.355372 32655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:29:13.355619 32641 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:29:13.355739 32641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576899143097-23430-0/minicluster-data/master-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576899143097-23430-0/minicluster-data/master-0/wal
uuid: "101546e0c10f4093989716a4818eb766"
format_stamp: "Formatted at 2026-09-28 06:29:13 on dist-test-slave-d59m"
server_key: "02f2938c0cd3c0c27c4faa31c0a97993"
server_key_iv: "cca2997193d060011f0fec64fde565bc"
server_key_version: "encryptionkey@0"
I20260928 06:29:13.355842 32641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576899143097-23430-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576899143097-23430-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576899143097-23430-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:29:13.379007 32641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:29:13.379381 32641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:29:13.379545 32641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:29:13.383826 32641 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.190:40563
I20260928 06:29:13.383873 32707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.190:40563 every 8 connection(s)
I20260928 06:29:13.384248 32641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
I20260928 06:29:13.385069 32708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:29:13.387677 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 32641
I20260928 06:29:13.387797 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576899143097-23430-0/minicluster-data/master-0/wal/instance
I20260928 06:29:13.388026 23430 external_mini_cluster.cc:1428] Setting key 28d8b9a626f9eae85665801bea8353b9
I20260928 06:29:13.388562 32708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 101546e0c10f4093989716a4818eb766: Bootstrap starting.
I20260928 06:29:13.389313 32708 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 101546e0c10f4093989716a4818eb766: Neither blocks nor log segments found. Creating new log.
I20260928 06:29:13.389648 32708 log.cc:824] T 00000000000000000000000000000000 P 101546e0c10f4093989716a4818eb766: Log is configured to *not* fsync() on all Append() calls
I20260928 06:29:13.390491 32708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 101546e0c10f4093989716a4818eb766: No bootstrap required, opened a new log
I20260928 06:29:13.392627 32708 raft_consensus.cc:348] T 00000000000000000000000000000000 P 101546e0c10f4093989716a4818eb766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "101546e0c10f4093989716a4818eb766" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 40563 } }
I20260928 06:29:13.392797 32708 raft_consensus.cc:374] T 00000000000000000000000000000000 P 101546e0c10f4093989716a4818eb766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:29:13.392831 32708 raft_consensus.cc:729] T 00000000000000000000000000000000 P 101546e0c10f4093989716a4818eb766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 101546e0c10f4093989716a4818eb766, State: Initialized, Role: FOLLOWER
I20260928 06:29:13.392951 32708 consensus_queue.cc:260] T 00000000000000000000000000000000 P 101546e0c10f4093989716a4818eb766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "101546e0c10f4093989716a4818eb766" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 40563 } }
I20260928 06:29:13.393044 32708 raft_consensus.cc:388] T 00000000000000000000000000000000 P 101546e0c10f4093989716a4818eb766 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:29:13.393090 32708 raft_consensus.cc:482] T 00000000000000000000000000000000 P 101546e0c10f4093989716a4818eb766 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:29:13.393131 32708 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 101546e0c10f4093989716a4818eb766 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:29:13.393821 32708 raft_consensus.cc:504] T 00000000000000000000000000000000 P 101546e0c10f4093989716a4818eb766 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "101546e0c10f4093989716a4818eb766" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 40563 } }
I20260928 06:29:13.393953 32708 leader_election.cc:302] T 00000000000000000000000000000000 P 101546e0c10f4093989716a4818eb766 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 101546e0c10f4093989716a4818eb766; no voters:
I20260928 06:29:13.394122 32708 leader_election.cc:288] T 00000000000000000000000000000000 P 101546e0c10f4093989716a4818eb766 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:29:13.394534 32708 sys_catalog.cc:564] T 00000000000000000000000000000000 P 101546e0c10f4093989716a4818eb766 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:29:13.394621 32711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:13.388571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59850 (local address 127.22.225.190:40563)
0928 06:29:13.388795 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 06:29:13.388829 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0928 06:29:13.388855 (+ 26us) server_negotiation.cc:407] Connection header received
0928 06:29:13.389475 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:29:13.389484 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:29:13.389584 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:29:13.389740 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:29:13.389962 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:13.391316 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:13.392371 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:13.393015 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:13.393138 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:13.393917 (+ 779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:29:13.393922 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:29:13.393922 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:29:13.394140 (+ 218us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:29:13.394186 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:29:13.394202 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:29:13.394270 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 06:29:13.394351 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 06:29:13.394434 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":67,"threads_started":1}
I20260928 06:29:13.394178 32713 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 101546e0c10f4093989716a4818eb766 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:29:13.395977 32713 raft_consensus.cc:686] T 00000000000000000000000000000000 P 101546e0c10f4093989716a4818eb766 [term 1 LEADER]: Becoming Leader. State: Replica: 101546e0c10f4093989716a4818eb766, State: Running, Role: LEADER
I20260928 06:29:13.396099 32713 consensus_queue.cc:237] T 00000000000000000000000000000000 P 101546e0c10f4093989716a4818eb766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "101546e0c10f4093989716a4818eb766" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 40563 } }
I20260928 06:29:13.396488 32725 sys_catalog.cc:455] T 00000000000000000000000000000000 P 101546e0c10f4093989716a4818eb766 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 101546e0c10f4093989716a4818eb766. Latest consensus state: current_term: 1 leader_uuid: "101546e0c10f4093989716a4818eb766" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "101546e0c10f4093989716a4818eb766" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 40563 } } }
I20260928 06:29:13.396709 32725 sys_catalog.cc:458] T 00000000000000000000000000000000 P 101546e0c10f4093989716a4818eb766 [sys.catalog]: This master's current role is: LEADER
I20260928 06:29:13.396694 32714 sys_catalog.cc:455] T 00000000000000000000000000000000 P 101546e0c10f4093989716a4818eb766 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "101546e0c10f4093989716a4818eb766" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "101546e0c10f4093989716a4818eb766" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 40563 } } }
I20260928 06:29:13.396787 32714 sys_catalog.cc:458] T 00000000000000000000000000000000 P 101546e0c10f4093989716a4818eb766 [sys.catalog]: This master's current role is: LEADER
I20260928 06:29:13.397064 32727 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:29:13.397747 32727 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:29:13.399367 32727 catalog_manager.cc:1347] Generated new cluster ID: f6f0c7390dde4163a8d98e85cfc0fc4c
I20260928 06:29:13.399410 32727 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:29:13.430430 32727 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:29:13.431116 32727 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:29:13.439409 32727 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 101546e0c10f4093989716a4818eb766: Generated new TSK 0
I20260928 06:29:13.439602 32727 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:29:13.449447 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.129:0
--local_ip_for_outbound_sockets=127.22.225.129
--webserver_interface=127.22.225.129
--webserver_port=0
--tserver_master_addrs=127.22.225.190:40563
--builtin_ntp_servers=127.22.225.148:34683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 06:29:13.565287 32732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:29:13.565485 32732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:29:13.565505 32732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:29:13.566668 32732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:29:13.566709 32732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:29:13.566733 32732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.129
I20260928 06:29:13.568075 32732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:34683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.225.129
--webserver_port=0
--tserver_master_addrs=127.22.225.190:40563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.129
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:29:13.568320 32732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:29:13.568668 32732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:29:13.573443 32738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:29:13.573442 32737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:29:13.573673 32732 server_base.cc:1034] running on GCE node
W20260928 06:29:13.573459 32740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:29:13.573969 32732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:29:13.574229 32732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:29:13.575399 32732 hybrid_clock.cc:648] HybridClock initialized: now 1790576953575394 us; error 31 us; skew 500 ppm
I20260928 06:29:13.576843 32732 webserver.cc:460] Webserver started at http://127.22.225.129:40625/ using document root <none> and password file <none>
I20260928 06:29:13.577091 32732 fs_manager.cc:359] Metadata directory not provided
I20260928 06:29:13.577135 32732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:29:13.577270 32732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:29:13.578318 32732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-0/data/instance:
uuid: "d3bc2e63214f4bc484fd66bda1eb76ad"
format_stamp: "Formatted at 2026-09-28 06:29:13 on dist-test-slave-d59m"
server_key: "fb5236a0900645ec2456d9c6bcb203bf"
server_key_iv: "b178075444c0a17d36bec274f52ba4f1"
server_key_version: "encryptionkey@0"
I20260928 06:29:13.578681 32732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-0/wal/instance:
uuid: "d3bc2e63214f4bc484fd66bda1eb76ad"
format_stamp: "Formatted at 2026-09-28 06:29:13 on dist-test-slave-d59m"
server_key: "fb5236a0900645ec2456d9c6bcb203bf"
server_key_iv: "b178075444c0a17d36bec274f52ba4f1"
server_key_version: "encryptionkey@0"
I20260928 06:29:13.579977 32732 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:29:13.580797 32746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:29:13.580964 32732 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:29:13.581048 32732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-0/wal
uuid: "d3bc2e63214f4bc484fd66bda1eb76ad"
format_stamp: "Formatted at 2026-09-28 06:29:13 on dist-test-slave-d59m"
server_key: "fb5236a0900645ec2456d9c6bcb203bf"
server_key_iv: "b178075444c0a17d36bec274f52ba4f1"
server_key_version: "encryptionkey@0"
I20260928 06:29:13.581117 32732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:29:13.610059 32732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:29:13.610394 32732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:29:13.610553 32732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:29:13.610832 32732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:29:13.611207 32732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:29:13.611243 32732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:29:13.611265 32732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:29:13.611307 32732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:29:13.618716 32732 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.129:46673
I20260928 06:29:13.618748 391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.129:46673 every 8 connection(s)
I20260928 06:29:13.619117 32732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
I20260928 06:29:13.625540 32711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:13.620026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:37763 (local address 127.22.225.190:40563)
0928 06:29:13.620098 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 06:29:13.620100 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:29:13.620422 (+ 322us) server_negotiation.cc:407] Connection header received
0928 06:29:13.621167 (+ 745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:29:13.621168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:29:13.621182 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:29:13.621230 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:29:13.622121 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:13.622622 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:13.623982 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:13.624176 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:13.624227 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:13.625221 (+ 994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:29:13.625222 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:29:13.625222 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:29:13.625226 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:29:13.625243 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:29:13.625247 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:29:13.625295 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 06:29:13.625408 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 06:29:13.625440 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 06:29:13.625588 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 32732
I20260928 06:29:13.625669 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-0/wal/instance
I20260928 06:29:13.625725 394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:13.620015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:40563 (local address 127.22.225.129:37763)
0928 06:29:13.620267 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0928 06:29:13.620319 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 06:29:13.621041 (+ 722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:29:13.621331 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:29:13.621336 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:29:13.621348 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:29:13.622029 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:13.622036 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:13.622717 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:13.622718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:13.623887 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:13.623890 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:13.624258 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:13.624259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:13.624412 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:13.625052 (+ 640us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:29:13.625052 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:29:13.625092 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:29:13.625122 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:29:13.625312 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:29:13.625318 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:29:13.625321 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:29:13.625391 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 06:29:13.625503 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":113,"threads_started":1}
I20260928 06:29:13.625892 23430 external_mini_cluster.cc:1428] Setting key d1781c8aba2c6fc60e7cf3ec96982995
I20260928 06:29:13.626371 392 heartbeater.cc:347] Connected to a master server at 127.22.225.190:40563
I20260928 06:29:13.626502 392 heartbeater.cc:464] Registering TS with master...
I20260928 06:29:13.626690 392 heartbeater.cc:511] Master 127.22.225.190:40563 requested a full tablet report, sending...
I20260928 06:29:13.626901 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.130:0
--local_ip_for_outbound_sockets=127.22.225.130
--webserver_interface=127.22.225.130
--webserver_port=0
--tserver_master_addrs=127.22.225.190:40563
--builtin_ntp_servers=127.22.225.148:34683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 06:29:13.627256 32672 ts_manager.cc:194] Registered new tserver with Master: d3bc2e63214f4bc484fd66bda1eb76ad (127.22.225.129:46673)
I20260928 06:29:13.627872 32672 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.225.129:37763
W20260928 06:29:13.732378 395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:29:13.732563 395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:29:13.732661 395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:29:13.733767 395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:29:13.733803 395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:29:13.733826 395 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.130
I20260928 06:29:13.735113 395 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:34683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.225.130
--webserver_port=0
--tserver_master_addrs=127.22.225.190:40563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.395
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.130
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:29:13.735373 395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:29:13.735672 395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:29:13.740296 401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:29:13.740254 400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:29:13.740422 395 server_base.cc:1034] running on GCE node
W20260928 06:29:13.740254 403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:29:13.740780 395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:29:13.741060 395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:29:13.742201 395 hybrid_clock.cc:648] HybridClock initialized: now 1790576953742181 us; error 38 us; skew 500 ppm
I20260928 06:29:13.743396 395 webserver.cc:460] Webserver started at http://127.22.225.130:34653/ using document root <none> and password file <none>
I20260928 06:29:13.743624 395 fs_manager.cc:359] Metadata directory not provided
I20260928 06:29:13.743688 395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:29:13.743839 395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:29:13.744952 395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-1/data/instance:
uuid: "0acb65b017094d63b87ba80281db627e"
format_stamp: "Formatted at 2026-09-28 06:29:13 on dist-test-slave-d59m"
server_key: "422510caa5edab9a6066a98a3b082696"
server_key_iv: "27c8e277244dc7b9baf545d54be5c6d8"
server_key_version: "encryptionkey@0"
I20260928 06:29:13.745291 395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-1/wal/instance:
uuid: "0acb65b017094d63b87ba80281db627e"
format_stamp: "Formatted at 2026-09-28 06:29:13 on dist-test-slave-d59m"
server_key: "422510caa5edab9a6066a98a3b082696"
server_key_iv: "27c8e277244dc7b9baf545d54be5c6d8"
server_key_version: "encryptionkey@0"
I20260928 06:29:13.746610 395 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:29:13.747346 409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:29:13.747510 395 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:29:13.747603 395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-1/wal
uuid: "0acb65b017094d63b87ba80281db627e"
format_stamp: "Formatted at 2026-09-28 06:29:13 on dist-test-slave-d59m"
server_key: "422510caa5edab9a6066a98a3b082696"
server_key_iv: "27c8e277244dc7b9baf545d54be5c6d8"
server_key_version: "encryptionkey@0"
I20260928 06:29:13.747682 395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:29:13.762251 395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:29:13.762557 395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:29:13.762733 395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:29:13.762953 395 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:29:13.763275 395 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:29:13.763329 395 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:29:13.763378 395 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:29:13.763406 395 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:29:13.769655 395 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.130:39333
I20260928 06:29:13.769732 522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.130:39333 every 8 connection(s)
I20260928 06:29:13.770087 395 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
I20260928 06:29:13.772698 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 395
I20260928 06:29:13.772810 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-1/wal/instance
I20260928 06:29:13.773042 23430 external_mini_cluster.cc:1428] Setting key 680f3ae08fc781b04a4c83a011220cbc
I20260928 06:29:13.774091 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.131:0
--local_ip_for_outbound_sockets=127.22.225.131
--webserver_interface=127.22.225.131
--webserver_port=0
--tserver_master_addrs=127.22.225.190:40563
--builtin_ntp_servers=127.22.225.148:34683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 06:29:13.777658 32711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:13.770899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:40515 (local address 127.22.225.190:40563)
0928 06:29:13.770959 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 06:29:13.770961 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:29:13.771326 (+ 365us) server_negotiation.cc:407] Connection header received
0928 06:29:13.772074 (+ 748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:29:13.772075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:29:13.772085 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:29:13.772146 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:29:13.773072 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:13.773719 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:13.776124 (+ 2405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:13.776325 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:13.776414 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:13.777343 (+ 929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:29:13.777344 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:29:13.777345 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:29:13.777350 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:29:13.777365 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:29:13.777370 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:29:13.777422 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 06:29:13.777559 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 06:29:13.777582 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 06:29:13.777856 525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:13.770864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:40563 (local address 127.22.225.130:40515)
0928 06:29:13.771187 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0928 06:29:13.771230 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:29:13.771943 (+ 713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:29:13.772223 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:29:13.772227 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:29:13.772239 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:29:13.772952 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:13.772959 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:13.773798 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:13.773800 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:13.776027 (+ 2227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:13.776030 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:13.776424 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:13.776426 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:13.776565 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:13.777190 (+ 625us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:29:13.777190 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:29:13.777217 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:29:13.777239 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:29:13.777431 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:29:13.777436 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:29:13.777439 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:29:13.777508 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 06:29:13.777617 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":221,"threads_started":1}
I20260928 06:29:13.778285 523 heartbeater.cc:347] Connected to a master server at 127.22.225.190:40563
I20260928 06:29:13.778381 523 heartbeater.cc:464] Registering TS with master...
I20260928 06:29:13.778551 523 heartbeater.cc:511] Master 127.22.225.190:40563 requested a full tablet report, sending...
I20260928 06:29:13.778913 32672 ts_manager.cc:194] Registered new tserver with Master: 0acb65b017094d63b87ba80281db627e (127.22.225.130:39333)
I20260928 06:29:13.779273 32672 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.225.130:40515
W20260928 06:29:13.872022 526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:29:13.872211 526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:29:13.872231 526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:29:13.873443 526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:29:13.873507 526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:29:13.873533 526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.131
I20260928 06:29:13.874799 526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:34683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.225.131
--webserver_port=0
--tserver_master_addrs=127.22.225.190:40563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.131
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:29:13.875043 526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:29:13.875341 526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:29:13.880414 531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:29:13.880430 532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:29:13.880705 526 server_base.cc:1034] running on GCE node
W20260928 06:29:13.880686 534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:29:13.881003 526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:29:13.881261 526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:29:13.882418 526 hybrid_clock.cc:648] HybridClock initialized: now 1790576953882402 us; error 34 us; skew 500 ppm
I20260928 06:29:13.883792 526 webserver.cc:460] Webserver started at http://127.22.225.131:41545/ using document root <none> and password file <none>
I20260928 06:29:13.884116 526 fs_manager.cc:359] Metadata directory not provided
I20260928 06:29:13.884181 526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:29:13.884339 526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:29:13.885471 526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-2/data/instance:
uuid: "cbb93e68b8d1410ba829d406ad817fa3"
format_stamp: "Formatted at 2026-09-28 06:29:13 on dist-test-slave-d59m"
server_key: "58544e266c8ebc79a9c449430dc063ca"
server_key_iv: "6dc30bf0810dc14c096d904505856750"
server_key_version: "encryptionkey@0"
I20260928 06:29:13.885841 526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-2/wal/instance:
uuid: "cbb93e68b8d1410ba829d406ad817fa3"
format_stamp: "Formatted at 2026-09-28 06:29:13 on dist-test-slave-d59m"
server_key: "58544e266c8ebc79a9c449430dc063ca"
server_key_iv: "6dc30bf0810dc14c096d904505856750"
server_key_version: "encryptionkey@0"
I20260928 06:29:13.887149 526 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 06:29:13.887908 540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:29:13.888062 526 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:29:13.888164 526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-2/wal
uuid: "cbb93e68b8d1410ba829d406ad817fa3"
format_stamp: "Formatted at 2026-09-28 06:29:13 on dist-test-slave-d59m"
server_key: "58544e266c8ebc79a9c449430dc063ca"
server_key_iv: "6dc30bf0810dc14c096d904505856750"
server_key_version: "encryptionkey@0"
I20260928 06:29:13.888235 526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:29:13.916114 526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:29:13.916435 526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:29:13.916633 526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:29:13.916889 526 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:29:13.917260 526 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:29:13.917316 526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:29:13.917358 526 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:29:13.917385 526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:29:13.924463 526 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.131:40739
I20260928 06:29:13.924541 653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.131:40739 every 8 connection(s)
I20260928 06:29:13.924911 526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
I20260928 06:29:13.929553 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 526
I20260928 06:29:13.929641 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-2/wal/instance
I20260928 06:29:13.929860 23430 external_mini_cluster.cc:1428] Setting key 727e640c46a4965383ee636927ea49e0
I20260928 06:29:13.930647 32711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:13.925733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:51775 (local address 127.22.225.190:40563)
0928 06:29:13.925821 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0928 06:29:13.925823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:29:13.926039 (+ 216us) server_negotiation.cc:407] Connection header received
0928 06:29:13.926521 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:29:13.926522 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:29:13.926535 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:29:13.926617 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:29:13.927267 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:13.927953 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:13.929012 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:13.929266 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:13.929343 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:13.930348 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:29:13.930349 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:29:13.930350 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:29:13.930354 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:29:13.930368 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:29:13.930372 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:29:13.930434 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 06:29:13.930542 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 06:29:13.930568 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 06:29:13.931078 656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:13.925681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:40563 (local address 127.22.225.131:51775)
0928 06:29:13.925892 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 06:29:13.925935 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:29:13.926419 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:29:13.926672 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:29:13.926675 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:29:13.926684 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:29:13.927171 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:13.927175 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:13.928048 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:13.928048 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:13.928895 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:13.928921 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:13.929354 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:13.929355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:13.929538 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:13.930182 (+ 644us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:29:13.930183 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:29:13.930237 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:29:13.930265 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:29:13.930459 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:29:13.930466 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:29:13.930477 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 06:29:13.930545 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 06:29:13.930649 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":104,"threads_started":1}
I20260928 06:29:13.931506 654 heartbeater.cc:347] Connected to a master server at 127.22.225.190:40563
I20260928 06:29:13.931595 654 heartbeater.cc:464] Registering TS with master...
I20260928 06:29:13.931792 654 heartbeater.cc:511] Master 127.22.225.190:40563 requested a full tablet report, sending...
I20260928 06:29:13.932219 32672 ts_manager.cc:194] Registered new tserver with Master: cbb93e68b8d1410ba829d406ad817fa3 (127.22.225.131:40739)
I20260928 06:29:13.932715 32672 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.225.131:51775
I20260928 06:29:13.941725 23430 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:29:13.943907 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 32732
W20260928 06:29:13.943902 32711 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:29:13.943398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59860 (local address 127.22.225.190:40563)
0928 06:29:13.943463 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 06:29:13.943465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:29:13.943564 (+ 99us) server_negotiation.cc:407] Connection header received
0928 06:29:13.943596 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:29:13.943597 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:29:13.943608 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:29:13.943673 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:29:13.943832 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59860: BlockingRecv error: recv got EOF from 127.0.0.1:59860 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 06:29:13.949302 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 395
I20260928 06:29:13.954592 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 526
I20260928 06:29:13.959285 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 32641
2026-09-28T06:29:13Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (765 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T06:29:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:29:13Z Disabled control of system clock
I20260928 06:29:13.987577 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576899143097-23430-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576899143097-23430-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576899143097-23430-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:45627
--webserver_interface=127.22.225.190
--webserver_port=0
--builtin_ntp_servers=127.22.225.148:35819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.225.190:45627
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 06:29:14.080652 665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:29:14.080868 665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:29:14.080889 665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:29:14.082043 665 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:29:14.082085 665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:29:14.082098 665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:29:14.082110 665 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:29:14.082160 665 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:29:14.083457 665 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:35819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576899143097-23430-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576899143097-23430-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.225.190:45627
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:45627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576899143097-23430-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:29:14.083698 665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:29:14.083978 665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:29:14.088688 673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:29:14.088692 670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:29:14.088692 671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:29:14.088910 665 server_base.cc:1034] running on GCE node
I20260928 06:29:14.089250 665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:29:14.089570 665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:29:14.090737 665 hybrid_clock.cc:648] HybridClock initialized: now 1790576954090718 us; error 36 us; skew 500 ppm
I20260928 06:29:14.095091 665 webserver.cc:460] Webserver started at http://127.22.225.190:36883/ using document root <none> and password file <none>
I20260928 06:29:14.095348 665 fs_manager.cc:359] Metadata directory not provided
I20260928 06:29:14.095427 665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:29:14.095587 665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:29:14.096743 665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576899143097-23430-0/minicluster-data/master-0/data/instance:
uuid: "6997cda1e8af4cdd90e5348b3911e0a5"
format_stamp: "Formatted at 2026-09-28 06:29:14 on dist-test-slave-d59m"
server_key: "c9f01573a5937dd2cc2c0f525e80e2ed"
server_key_iv: "f9a5facb992772da5e7ef476fb15e990"
server_key_version: "encryptionkey@0"
I20260928 06:29:14.097110 665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576899143097-23430-0/minicluster-data/master-0/wal/instance:
uuid: "6997cda1e8af4cdd90e5348b3911e0a5"
format_stamp: "Formatted at 2026-09-28 06:29:14 on dist-test-slave-d59m"
server_key: "c9f01573a5937dd2cc2c0f525e80e2ed"
server_key_iv: "f9a5facb992772da5e7ef476fb15e990"
server_key_version: "encryptionkey@0"
I20260928 06:29:14.098423 665 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 06:29:14.099193 679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:29:14.099361 665 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:29:14.099460 665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576899143097-23430-0/minicluster-data/master-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576899143097-23430-0/minicluster-data/master-0/wal
uuid: "6997cda1e8af4cdd90e5348b3911e0a5"
format_stamp: "Formatted at 2026-09-28 06:29:14 on dist-test-slave-d59m"
server_key: "c9f01573a5937dd2cc2c0f525e80e2ed"
server_key_iv: "f9a5facb992772da5e7ef476fb15e990"
server_key_version: "encryptionkey@0"
I20260928 06:29:14.099534 665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576899143097-23430-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576899143097-23430-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576899143097-23430-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:29:14.102954 665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:29:14.103238 665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:29:14.103379 665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:29:14.108127 665 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.190:45627
I20260928 06:29:14.108155 731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.190:45627 every 8 connection(s)
I20260928 06:29:14.108498 665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
I20260928 06:29:14.109267 732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:29:14.112022 732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6997cda1e8af4cdd90e5348b3911e0a5: Bootstrap starting.
I20260928 06:29:14.112094 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 665
I20260928 06:29:14.112179 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576899143097-23430-0/minicluster-data/master-0/wal/instance
I20260928 06:29:14.112351 23430 external_mini_cluster.cc:1428] Setting key e3da3f598fb957f8e606257874aac8c7
I20260928 06:29:14.112794 732 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6997cda1e8af4cdd90e5348b3911e0a5: Neither blocks nor log segments found. Creating new log.
I20260928 06:29:14.113204 732 log.cc:824] T 00000000000000000000000000000000 P 6997cda1e8af4cdd90e5348b3911e0a5: Log is configured to *not* fsync() on all Append() calls
I20260928 06:29:14.114002 732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6997cda1e8af4cdd90e5348b3911e0a5: No bootstrap required, opened a new log
I20260928 06:29:14.115039 735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:14.112854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39826 (local address 127.22.225.190:45627)
0928 06:29:14.113060 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 06:29:14.113068 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:29:14.113098 (+ 30us) server_negotiation.cc:407] Connection header received
0928 06:29:14.113586 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:29:14.113594 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:29:14.113641 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:29:14.113788 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:29:14.114587 (+ 799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:29:14.114590 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:29:14.114591 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:29:14.114595 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:29:14.114643 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:29:14.114656 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:29:14.114741 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 06:29:14.114805 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 06:29:14.114858 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":71,"threads_started":1}
I20260928 06:29:14.115953 732 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6997cda1e8af4cdd90e5348b3911e0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6997cda1e8af4cdd90e5348b3911e0a5" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 45627 } }
I20260928 06:29:14.116104 732 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6997cda1e8af4cdd90e5348b3911e0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:29:14.116150 732 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6997cda1e8af4cdd90e5348b3911e0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6997cda1e8af4cdd90e5348b3911e0a5, State: Initialized, Role: FOLLOWER
I20260928 06:29:14.116268 732 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6997cda1e8af4cdd90e5348b3911e0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6997cda1e8af4cdd90e5348b3911e0a5" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 45627 } }
I20260928 06:29:14.116364 732 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6997cda1e8af4cdd90e5348b3911e0a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:29:14.116407 732 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6997cda1e8af4cdd90e5348b3911e0a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:29:14.116456 732 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6997cda1e8af4cdd90e5348b3911e0a5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:29:14.117324 732 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6997cda1e8af4cdd90e5348b3911e0a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6997cda1e8af4cdd90e5348b3911e0a5" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 45627 } }
I20260928 06:29:14.117445 732 leader_election.cc:302] T 00000000000000000000000000000000 P 6997cda1e8af4cdd90e5348b3911e0a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6997cda1e8af4cdd90e5348b3911e0a5; no voters:
I20260928 06:29:14.117599 732 leader_election.cc:288] T 00000000000000000000000000000000 P 6997cda1e8af4cdd90e5348b3911e0a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:29:14.117645 737 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6997cda1e8af4cdd90e5348b3911e0a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:29:14.117786 737 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6997cda1e8af4cdd90e5348b3911e0a5 [term 1 LEADER]: Becoming Leader. State: Replica: 6997cda1e8af4cdd90e5348b3911e0a5, State: Running, Role: LEADER
I20260928 06:29:14.117908 732 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6997cda1e8af4cdd90e5348b3911e0a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:29:14.117926 737 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6997cda1e8af4cdd90e5348b3911e0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6997cda1e8af4cdd90e5348b3911e0a5" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 45627 } }
I20260928 06:29:14.118327 737 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6997cda1e8af4cdd90e5348b3911e0a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6997cda1e8af4cdd90e5348b3911e0a5. Latest consensus state: current_term: 1 leader_uuid: "6997cda1e8af4cdd90e5348b3911e0a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6997cda1e8af4cdd90e5348b3911e0a5" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 45627 } } }
I20260928 06:29:14.118444 737 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6997cda1e8af4cdd90e5348b3911e0a5 [sys.catalog]: This master's current role is: LEADER
I20260928 06:29:14.118342 738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6997cda1e8af4cdd90e5348b3911e0a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6997cda1e8af4cdd90e5348b3911e0a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6997cda1e8af4cdd90e5348b3911e0a5" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 45627 } } }
I20260928 06:29:14.118585 738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6997cda1e8af4cdd90e5348b3911e0a5 [sys.catalog]: This master's current role is: LEADER
I20260928 06:29:14.118806 744 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:29:14.119326 744 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:29:14.125797 744 catalog_manager.cc:1347] Generated new cluster ID: c68a49e310474d31a46b39ec5934d5fd
I20260928 06:29:14.125869 744 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:29:14.142313 744 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:29:14.142511 744 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:29:14.155869 744 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6997cda1e8af4cdd90e5348b3911e0a5: Generated new TSK 0
I20260928 06:29:14.156090 744 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:29:14.162637 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.129:0
--local_ip_for_outbound_sockets=127.22.225.129
--webserver_interface=127.22.225.129
--webserver_port=0
--tserver_master_addrs=127.22.225.190:45627
--builtin_ntp_servers=127.22.225.148:35819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 06:29:14.247900 756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:29:14.248128 756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:29:14.248163 756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:29:14.249370 756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:29:14.249433 756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:29:14.249473 756 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.129
I20260928 06:29:14.250821 756 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:35819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.225.129
--webserver_port=0
--tserver_master_addrs=127.22.225.190:45627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.756
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.129
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:29:14.251087 756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:29:14.251395 756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:29:14.256371 762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:29:14.256400 764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:29:14.256764 756 server_base.cc:1034] running on GCE node
W20260928 06:29:14.256615 761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:29:14.257169 756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:29:14.257433 756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:29:14.258605 756 hybrid_clock.cc:648] HybridClock initialized: now 1790576954258593 us; error 38 us; skew 500 ppm
I20260928 06:29:14.259845 756 webserver.cc:460] Webserver started at http://127.22.225.129:37923/ using document root <none> and password file <none>
I20260928 06:29:14.260124 756 fs_manager.cc:359] Metadata directory not provided
I20260928 06:29:14.260198 756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:29:14.260349 756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:29:14.261500 756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-0/data/instance:
uuid: "bffcf90dd0a34802aa4adf3878aa57d3"
format_stamp: "Formatted at 2026-09-28 06:29:14 on dist-test-slave-d59m"
server_key: "2e6feee1b045ef65f5f6933df4aaece4"
server_key_iv: "30a85f9f5706d459f35845a81bfa8261"
server_key_version: "encryptionkey@0"
I20260928 06:29:14.261871 756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-0/wal/instance:
uuid: "bffcf90dd0a34802aa4adf3878aa57d3"
format_stamp: "Formatted at 2026-09-28 06:29:14 on dist-test-slave-d59m"
server_key: "2e6feee1b045ef65f5f6933df4aaece4"
server_key_iv: "30a85f9f5706d459f35845a81bfa8261"
server_key_version: "encryptionkey@0"
I20260928 06:29:14.263213 756 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:29:14.264030 770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:29:14.264189 756 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:29:14.264273 756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-0/wal
uuid: "bffcf90dd0a34802aa4adf3878aa57d3"
format_stamp: "Formatted at 2026-09-28 06:29:14 on dist-test-slave-d59m"
server_key: "2e6feee1b045ef65f5f6933df4aaece4"
server_key_iv: "30a85f9f5706d459f35845a81bfa8261"
server_key_version: "encryptionkey@0"
I20260928 06:29:14.264343 756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:29:14.267731 756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:29:14.268000 756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:29:14.268149 756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:29:14.268424 756 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:29:14.268867 756 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:29:14.268920 756 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:29:14.268954 756 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:29:14.268982 756 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:29:14.276343 756 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.129:45509
I20260928 06:29:14.276379 883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.129:45509 every 8 connection(s)
I20260928 06:29:14.276782 756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
I20260928 06:29:14.278036 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 756
I20260928 06:29:14.278137 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-0/wal/instance
I20260928 06:29:14.278321 23430 external_mini_cluster.cc:1428] Setting key 0445c4cb9a6fc54fdfdcb917de80c6ce
I20260928 06:29:14.279706 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.130:0
--local_ip_for_outbound_sockets=127.22.225.130
--webserver_interface=127.22.225.130
--webserver_port=0
--tserver_master_addrs=127.22.225.190:45627
--builtin_ntp_servers=127.22.225.148:35819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 06:29:14.280831 735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:14.277524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:39219 (local address 127.22.225.190:45627)
0928 06:29:14.277570 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 06:29:14.277572 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:29:14.277920 (+ 348us) server_negotiation.cc:407] Connection header received
0928 06:29:14.278652 (+ 732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:29:14.278653 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:29:14.278660 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:29:14.278715 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:29:14.280492 (+ 1777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:29:14.280493 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:29:14.280494 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:29:14.280499 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:29:14.280516 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:29:14.280521 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:29:14.280552 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0928 06:29:14.280716 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 06:29:14.280734 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 06:29:14.281379 886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:14.277543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:45627 (local address 127.22.225.129:39219)
0928 06:29:14.277752 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 06:29:14.277801 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 06:29:14.278402 (+ 601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:29:14.278845 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:29:14.278850 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:29:14.278859 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:29:14.280272 (+ 1413us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:29:14.280273 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:29:14.280312 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:29:14.280349 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:29:14.280621 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:29:14.280627 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:29:14.280630 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:29:14.281070 (+ 440us) client_negotiation.cc:240] Negotiation successful
0928 06:29:14.281148 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":99,"threads_started":1}
I20260928 06:29:14.281840 884 heartbeater.cc:347] Connected to a master server at 127.22.225.190:45627
I20260928 06:29:14.281935 884 heartbeater.cc:464] Registering TS with master...
I20260928 06:29:14.282099 884 heartbeater.cc:511] Master 127.22.225.190:45627 requested a full tablet report, sending...
I20260928 06:29:14.282609 696 ts_manager.cc:194] Registered new tserver with Master: bffcf90dd0a34802aa4adf3878aa57d3 (127.22.225.129:45509)
W20260928 06:29:14.376327 887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:29:14.376672 887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:29:14.376740 887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:29:14.378582 887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:29:14.378664 887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:29:14.378710 887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.130
I20260928 06:29:14.380784 887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:35819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.225.130
--webserver_port=0
--tserver_master_addrs=127.22.225.190:45627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.130
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:29:14.381075 887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:29:14.381390 887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:29:14.386519 893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:29:14.386534 892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:29:14.386534 895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:29:14.386937 887 server_base.cc:1034] running on GCE node
I20260928 06:29:14.387171 887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:29:14.387463 887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:29:14.388651 887 hybrid_clock.cc:648] HybridClock initialized: now 1790576954388631 us; error 33 us; skew 500 ppm
I20260928 06:29:14.390188 887 webserver.cc:460] Webserver started at http://127.22.225.130:41231/ using document root <none> and password file <none>
I20260928 06:29:14.390482 887 fs_manager.cc:359] Metadata directory not provided
I20260928 06:29:14.390554 887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:29:14.390693 887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:29:14.392151 887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-1/data/instance:
uuid: "e96d73071a5b4ca59efd431c9de7f7e7"
format_stamp: "Formatted at 2026-09-28 06:29:14 on dist-test-slave-d59m"
server_key: "2487d4b8be21276fedeae6d154a31627"
server_key_iv: "acba92685c95ce01e4709203773aa99b"
server_key_version: "encryptionkey@0"
I20260928 06:29:14.392530 887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-1/wal/instance:
uuid: "e96d73071a5b4ca59efd431c9de7f7e7"
format_stamp: "Formatted at 2026-09-28 06:29:14 on dist-test-slave-d59m"
server_key: "2487d4b8be21276fedeae6d154a31627"
server_key_iv: "acba92685c95ce01e4709203773aa99b"
server_key_version: "encryptionkey@0"
I20260928 06:29:14.394381 887 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260928 06:29:14.395325 901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:29:14.395498 887 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:29:14.395576 887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-1/wal
uuid: "e96d73071a5b4ca59efd431c9de7f7e7"
format_stamp: "Formatted at 2026-09-28 06:29:14 on dist-test-slave-d59m"
server_key: "2487d4b8be21276fedeae6d154a31627"
server_key_iv: "acba92685c95ce01e4709203773aa99b"
server_key_version: "encryptionkey@0"
I20260928 06:29:14.395674 887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:29:14.399061 887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:29:14.399328 887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:29:14.399473 887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:29:14.399683 887 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:29:14.399996 887 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:29:14.400053 887 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:29:14.400091 887 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:29:14.400118 887 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:29:14.407503 887 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.130:44035
I20260928 06:29:14.407547 1014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.130:44035 every 8 connection(s)
I20260928 06:29:14.407963 887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
I20260928 06:29:14.411918 735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:14.409662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:57073 (local address 127.22.225.190:45627)
0928 06:29:14.409731 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 06:29:14.409734 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:29:14.409747 (+ 13us) server_negotiation.cc:407] Connection header received
0928 06:29:14.409916 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:29:14.409917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:29:14.409923 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:29:14.409990 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:29:14.411543 (+ 1553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:29:14.411545 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:29:14.411546 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:29:14.411551 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:29:14.411571 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:29:14.411576 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:29:14.411641 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 06:29:14.411810 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 06:29:14.411833 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 06:29:14.412305 1017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:14.408879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:45627 (local address 127.22.225.130:57073)
0928 06:29:14.409106 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0928 06:29:14.409154 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 06:29:14.409806 (+ 652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:29:14.410119 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:29:14.410123 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:29:14.410133 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:29:14.411342 (+ 1209us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:29:14.411343 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:29:14.411382 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:29:14.411417 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:29:14.411673 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:29:14.411680 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:29:14.411683 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:29:14.411795 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 06:29:14.411863 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":101,"threads_started":1}
I20260928 06:29:14.412775 1015 heartbeater.cc:347] Connected to a master server at 127.22.225.190:45627
I20260928 06:29:14.412870 1015 heartbeater.cc:464] Registering TS with master...
I20260928 06:29:14.413029 1015 heartbeater.cc:511] Master 127.22.225.190:45627 requested a full tablet report, sending...
I20260928 06:29:14.413429 696 ts_manager.cc:194] Registered new tserver with Master: e96d73071a5b4ca59efd431c9de7f7e7 (127.22.225.130:44035)
I20260928 06:29:14.415488 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 887
I20260928 06:29:14.415591 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-1/wal/instance
I20260928 06:29:14.415802 23430 external_mini_cluster.cc:1428] Setting key 0eadfe92940b0d45c7c0ccfb7e893c0d
I20260928 06:29:14.416817 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.131:0
--local_ip_for_outbound_sockets=127.22.225.131
--webserver_interface=127.22.225.131
--webserver_port=0
--tserver_master_addrs=127.22.225.190:45627
--builtin_ntp_servers=127.22.225.148:35819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 06:29:14.501494 1018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:29:14.501756 1018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:29:14.501796 1018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:29:14.503050 1018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:29:14.503105 1018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:29:14.503147 1018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.131
I20260928 06:29:14.504501 1018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:35819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.225.131
--webserver_port=0
--tserver_master_addrs=127.22.225.190:45627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.131
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:29:14.504781 1018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:29:14.505076 1018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:29:14.509932 1026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:29:14.510010 1024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:29:14.510133 1018 server_base.cc:1034] running on GCE node
W20260928 06:29:14.509945 1023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:29:14.510389 1018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:29:14.510661 1018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:29:14.511821 1018 hybrid_clock.cc:648] HybridClock initialized: now 1790576954511807 us; error 30 us; skew 500 ppm
I20260928 06:29:14.513084 1018 webserver.cc:460] Webserver started at http://127.22.225.131:41847/ using document root <none> and password file <none>
I20260928 06:29:14.513325 1018 fs_manager.cc:359] Metadata directory not provided
I20260928 06:29:14.513396 1018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:29:14.513556 1018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:29:14.514568 1018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-2/data/instance:
uuid: "c760cbb594314c858e1f7729bce69554"
format_stamp: "Formatted at 2026-09-28 06:29:14 on dist-test-slave-d59m"
server_key: "c1b6ea09eebc5f396cdce0dedab9ce7b"
server_key_iv: "85fb794ef69ac753ee6568ab18d1c212"
server_key_version: "encryptionkey@0"
I20260928 06:29:14.514927 1018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-2/wal/instance:
uuid: "c760cbb594314c858e1f7729bce69554"
format_stamp: "Formatted at 2026-09-28 06:29:14 on dist-test-slave-d59m"
server_key: "c1b6ea09eebc5f396cdce0dedab9ce7b"
server_key_iv: "85fb794ef69ac753ee6568ab18d1c212"
server_key_version: "encryptionkey@0"
I20260928 06:29:14.516245 1018 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:29:14.517113 1032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:29:14.517283 1018 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:29:14.517381 1018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-2/wal
uuid: "c760cbb594314c858e1f7729bce69554"
format_stamp: "Formatted at 2026-09-28 06:29:14 on dist-test-slave-d59m"
server_key: "c1b6ea09eebc5f396cdce0dedab9ce7b"
server_key_iv: "85fb794ef69ac753ee6568ab18d1c212"
server_key_version: "encryptionkey@0"
I20260928 06:29:14.517465 1018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:29:14.521034 1018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:29:14.521313 1018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:29:14.521476 1018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:29:14.521687 1018 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:29:14.522019 1018 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:29:14.522063 1018 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:29:14.522087 1018 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:29:14.522110 1018 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:29:14.528846 1018 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.131:42357
I20260928 06:29:14.528878 1145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.131:42357 every 8 connection(s)
I20260928 06:29:14.529250 1018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
I20260928 06:29:14.531605 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 1018
I20260928 06:29:14.531713 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576899143097-23430-0/minicluster-data/ts-2/wal/instance
I20260928 06:29:14.531975 23430 external_mini_cluster.cc:1428] Setting key eb9cc023c496751346f6caf4f093e451
I20260928 06:29:14.532477 735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:14.529965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:50881 (local address 127.22.225.190:45627)
0928 06:29:14.530013 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 06:29:14.530015 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:29:14.530246 (+ 231us) server_negotiation.cc:407] Connection header received
0928 06:29:14.530690 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:29:14.530692 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:29:14.530699 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:29:14.530778 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:29:14.532212 (+ 1434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:29:14.532213 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:29:14.532214 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:29:14.532219 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:29:14.532236 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:29:14.532240 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:29:14.532292 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 06:29:14.532390 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 06:29:14.532406 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:29:14.532718 1148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:14.529925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:45627 (local address 127.22.225.131:50881)
0928 06:29:14.530109 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 06:29:14.530148 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 06:29:14.530595 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:29:14.530844 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:29:14.530848 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:29:14.530857 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:29:14.532032 (+ 1175us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:29:14.532033 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:29:14.532073 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:29:14.532109 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:29:14.532290 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:29:14.532295 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:29:14.532298 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:29:14.532398 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 06:29:14.532461 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":93,"threads_started":1}
I20260928 06:29:14.533190 1146 heartbeater.cc:347] Connected to a master server at 127.22.225.190:45627
I20260928 06:29:14.533278 1146 heartbeater.cc:464] Registering TS with master...
I20260928 06:29:14.533387 1146 heartbeater.cc:511] Master 127.22.225.190:45627 requested a full tablet report, sending...
I20260928 06:29:14.533718 696 ts_manager.cc:194] Registered new tserver with Master: c760cbb594314c858e1f7729bce69554 (127.22.225.131:42357)
I20260928 06:29:14.544147 23430 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 06:29:14.546275 735 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:29:14.545738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39838 (local address 127.22.225.190:45627)
0928 06:29:14.545822 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 06:29:14.545825 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:29:14.545954 (+ 129us) server_negotiation.cc:407] Connection header received
0928 06:29:14.546006 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:29:14.546007 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:29:14.546013 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:29:14.546075 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:29:14.546203 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39838: BlockingRecv error: recv got EOF from 127.0.0.1:39838 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 06:29:14.546306 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 756
I20260928 06:29:14.551666 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 887
I20260928 06:29:14.556296 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 1018
I20260928 06:29:14.560235 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 665
2026-09-28T06:29:14Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (600 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:29:14 dist-test-slave-d59m krb5kdc[1156](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:29:14 dist-test-slave-d59m krb5kdc[1156](info): set up 2 sockets
Sep 28 06:29:14 dist-test-slave-d59m krb5kdc[1156](info): commencing operation
krb5kdc: starting...
W20260928 06:29:15.662155 23430 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Sep 28 06:29:15 dist-test-slave-d59m krb5kdc[1156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:29:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:29:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab.
Entry for principal kudu/127.22.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab.
I20260928 06:29:15.793489 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:33009
--webserver_interface=127.22.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab
--principal=kudu/127.22.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:41577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.225.190:33009
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:29:15.892105 1169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:29:15.892310 1169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:29:15.892331 1169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:29:15.893528 1169 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:29:15.893568 1169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:29:15.893582 1169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:29:15.893594 1169 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:29:15.893606 1169 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:29:15.895174 1169 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:41577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.225.190:33009
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab
--principal=kudu/127.22.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:33009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:29:15.895417 1169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:29:15.895681 1169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:29:15.900553 1174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:29:15.900553 1177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:29:15.900612 1175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:29:15.900867 1169 server_base.cc:1034] running on GCE node
I20260928 06:29:15.901136 1169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:29:15.901449 1169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:29:15.902611 1169 hybrid_clock.cc:648] HybridClock initialized: now 1790576955902583 us; error 50 us; skew 500 ppm
Sep 28 06:29:15 dist-test-slave-d59m krb5kdc[1156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576955, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:29:15.905167 1169 init.cc:377] Logged in from keytab as kudu/127.22.225.190@KRBTEST.COM (short username kudu)
I20260928 06:29:15.905635 1169 webserver.cc:460] Webserver started at http://127.22.225.190:41745/ using document root <none> and password file <none>
I20260928 06:29:15.905892 1169 fs_manager.cc:359] Metadata directory not provided
I20260928 06:29:15.905968 1169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:29:15.906121 1169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:29:15.907156 1169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/minicluster-data/master-0/data/instance:
uuid: "a42ae672dbb847cabc5a75452074b9a3"
format_stamp: "Formatted at 2026-09-28 06:29:15 on dist-test-slave-d59m"
server_key: "03ea99ea3600ace6d4189de46c1ad4b7"
server_key_iv: "3101cce27b04accdc040cbae08f7fe42"
server_key_version: "encryptionkey@0"
I20260928 06:29:15.907516 1169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/minicluster-data/master-0/wal/instance:
uuid: "a42ae672dbb847cabc5a75452074b9a3"
format_stamp: "Formatted at 2026-09-28 06:29:15 on dist-test-slave-d59m"
server_key: "03ea99ea3600ace6d4189de46c1ad4b7"
server_key_iv: "3101cce27b04accdc040cbae08f7fe42"
server_key_version: "encryptionkey@0"
I20260928 06:29:15.908874 1169 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 06:29:15.909665 1184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:29:15.909909 1169 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:29:15.910014 1169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/minicluster-data/master-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/minicluster-data/master-0/wal
uuid: "a42ae672dbb847cabc5a75452074b9a3"
format_stamp: "Formatted at 2026-09-28 06:29:15 on dist-test-slave-d59m"
server_key: "03ea99ea3600ace6d4189de46c1ad4b7"
server_key_iv: "3101cce27b04accdc040cbae08f7fe42"
server_key_version: "encryptionkey@0"
I20260928 06:29:15.910100 1169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:29:15.926551 1169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:29:15.928745 1169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:29:15.928944 1169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:29:15.933403 1169 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.190:33009
I20260928 06:29:15.933431 1236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.190:33009 every 8 connection(s)
I20260928 06:29:15.933797 1169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
I20260928 06:29:15.934806 1237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:29:15.937157 1237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a42ae672dbb847cabc5a75452074b9a3: Bootstrap starting.
I20260928 06:29:15.937759 1237 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a42ae672dbb847cabc5a75452074b9a3: Neither blocks nor log segments found. Creating new log.
I20260928 06:29:15.938068 1237 log.cc:824] T 00000000000000000000000000000000 P a42ae672dbb847cabc5a75452074b9a3: Log is configured to *not* fsync() on all Append() calls
I20260928 06:29:15.938812 1237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a42ae672dbb847cabc5a75452074b9a3: No bootstrap required, opened a new log
I20260928 06:29:15.939188 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 1169
I20260928 06:29:15.939298 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/minicluster-data/master-0/wal/instance
I20260928 06:29:15.939493 23430 external_mini_cluster.cc:1428] Setting key 29c0b3c01c2a86ccfe32b7ce4630fe9d
I20260928 06:29:15.941073 1237 raft_consensus.cc:348] T 00000000000000000000000000000000 P a42ae672dbb847cabc5a75452074b9a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a42ae672dbb847cabc5a75452074b9a3" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 33009 } }
I20260928 06:29:15.941244 1237 raft_consensus.cc:374] T 00000000000000000000000000000000 P a42ae672dbb847cabc5a75452074b9a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:29:15.941290 1237 raft_consensus.cc:729] T 00000000000000000000000000000000 P a42ae672dbb847cabc5a75452074b9a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a42ae672dbb847cabc5a75452074b9a3, State: Initialized, Role: FOLLOWER
I20260928 06:29:15.941437 1237 consensus_queue.cc:260] T 00000000000000000000000000000000 P a42ae672dbb847cabc5a75452074b9a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a42ae672dbb847cabc5a75452074b9a3" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 33009 } }
I20260928 06:29:15.941524 1237 raft_consensus.cc:388] T 00000000000000000000000000000000 P a42ae672dbb847cabc5a75452074b9a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:29:15.941562 1237 raft_consensus.cc:482] T 00000000000000000000000000000000 P a42ae672dbb847cabc5a75452074b9a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:29:15.941601 1237 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a42ae672dbb847cabc5a75452074b9a3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:29:15.942658 1237 raft_consensus.cc:504] T 00000000000000000000000000000000 P a42ae672dbb847cabc5a75452074b9a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a42ae672dbb847cabc5a75452074b9a3" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 33009 } }
I20260928 06:29:15.942762 1237 leader_election.cc:302] T 00000000000000000000000000000000 P a42ae672dbb847cabc5a75452074b9a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a42ae672dbb847cabc5a75452074b9a3; no voters:
I20260928 06:29:15.942921 1237 leader_election.cc:288] T 00000000000000000000000000000000 P a42ae672dbb847cabc5a75452074b9a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:29:15.943027 1242 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a42ae672dbb847cabc5a75452074b9a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:29:15.943133 1237 sys_catalog.cc:564] T 00000000000000000000000000000000 P a42ae672dbb847cabc5a75452074b9a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:29:15.943235 1242 raft_consensus.cc:686] T 00000000000000000000000000000000 P a42ae672dbb847cabc5a75452074b9a3 [term 1 LEADER]: Becoming Leader. State: Replica: a42ae672dbb847cabc5a75452074b9a3, State: Running, Role: LEADER
I20260928 06:29:15.943392 1242 consensus_queue.cc:237] T 00000000000000000000000000000000 P a42ae672dbb847cabc5a75452074b9a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a42ae672dbb847cabc5a75452074b9a3" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 33009 } }
I20260928 06:29:15.943787 1242 sys_catalog.cc:455] T 00000000000000000000000000000000 P a42ae672dbb847cabc5a75452074b9a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a42ae672dbb847cabc5a75452074b9a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a42ae672dbb847cabc5a75452074b9a3" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 33009 } } }
I20260928 06:29:15.943919 1242 sys_catalog.cc:458] T 00000000000000000000000000000000 P a42ae672dbb847cabc5a75452074b9a3 [sys.catalog]: This master's current role is: LEADER
I20260928 06:29:15.944079 1242 sys_catalog.cc:455] T 00000000000000000000000000000000 P a42ae672dbb847cabc5a75452074b9a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a42ae672dbb847cabc5a75452074b9a3. Latest consensus state: current_term: 1 leader_uuid: "a42ae672dbb847cabc5a75452074b9a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a42ae672dbb847cabc5a75452074b9a3" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 33009 } } }
I20260928 06:29:15.944170 1242 sys_catalog.cc:458] T 00000000000000000000000000000000 P a42ae672dbb847cabc5a75452074b9a3 [sys.catalog]: This master's current role is: LEADER
I20260928 06:29:15.944345 1251 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:29:15.944975 1251 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:29:15.946551 1251 catalog_manager.cc:1347] Generated new cluster ID: 2961400af54843eb8a56e190c3f23f43
I20260928 06:29:15.946599 1251 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:29:15 dist-test-slave-d59m krb5kdc[1156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:29:15.952695 1241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:15.940089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59850 (local address 127.22.225.190:33009)
0928 06:29:15.940706 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0928 06:29:15.940716 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:29:15.940755 (+ 39us) server_negotiation.cc:407] Connection header received
0928 06:29:15.941032 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:29:15.941041 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:29:15.941111 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:29:15.941426 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:29:15.942001 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:15.943422 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:15.944754 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:15.945379 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:15.945813 (+ 434us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:15.948510 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:29:15.948513 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:29:15.948514 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:29:15.948526 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:29:15.950552 (+ 2026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:29:15.950965 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:29:15.950968 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:29:15.950969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:29:15.951102 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:29:15.951320 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:29:15.951321 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:29:15.951322 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:29:15.951640 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:29:15.952216 (+ 576us) server_negotiation.cc:1035] Waiting for connection context
0928 06:29:15.952287 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 06:29:15.952445 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":89,"threads_started":1}
I20260928 06:29:15.957341 1251 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:29:15.957973 1251 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:29:15.973848 1251 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a42ae672dbb847cabc5a75452074b9a3: Generated new TSK 0
I20260928 06:29:15.974064 1251 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.225.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab.
Entry for principal kudu/127.22.225.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab.
I20260928 06:29:16.007961 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.129:0
--local_ip_for_outbound_sockets=127.22.225.129
--webserver_interface=127.22.225.129
--webserver_port=0
--tserver_master_addrs=127.22.225.190:33009
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab
--principal=kudu/127.22.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:41577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:29:16.124758 1263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:29:16.125005 1263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:29:16.125044 1263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:29:16.126240 1263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:29:16.126317 1263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:29:16.126372 1263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.129
I20260928 06:29:16.127833 1263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:41577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab
--principal=kudu/127.22.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.225.129
--webserver_port=0
--tserver_master_addrs=127.22.225.190:33009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1263
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.129
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:29:16.128114 1263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:29:16.128363 1263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:29:16.133473 1269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:29:16.133466 1271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:29:16.133471 1268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:29:16.133741 1263 server_base.cc:1034] running on GCE node
I20260928 06:29:16.134039 1263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:29:16.134301 1263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:29:16.135465 1263 hybrid_clock.cc:648] HybridClock initialized: now 1790576956135438 us; error 41 us; skew 500 ppm
Sep 28 06:29:16 dist-test-slave-d59m krb5kdc[1156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576956, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:29:16.138321 1263 init.cc:377] Logged in from keytab as kudu/127.22.225.129@KRBTEST.COM (short username kudu)
I20260928 06:29:16.138756 1263 webserver.cc:460] Webserver started at http://127.22.225.129:36879/ using document root <none> and password file <none>
I20260928 06:29:16.139000 1263 fs_manager.cc:359] Metadata directory not provided
I20260928 06:29:16.139076 1263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:29:16.139233 1263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:29:16.140282 1263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/minicluster-data/ts-0/data/instance:
uuid: "a4a6e4cf9f9d402db560b2202466ba1c"
format_stamp: "Formatted at 2026-09-28 06:29:16 on dist-test-slave-d59m"
server_key: "2753a2cbd13523c3ea40f54debdcb6ef"
server_key_iv: "a817370ce9a91dbacab290b67e21035a"
server_key_version: "encryptionkey@0"
I20260928 06:29:16.140722 1263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/minicluster-data/ts-0/wal/instance:
uuid: "a4a6e4cf9f9d402db560b2202466ba1c"
format_stamp: "Formatted at 2026-09-28 06:29:16 on dist-test-slave-d59m"
server_key: "2753a2cbd13523c3ea40f54debdcb6ef"
server_key_iv: "a817370ce9a91dbacab290b67e21035a"
server_key_version: "encryptionkey@0"
I20260928 06:29:16.142194 1263 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:29:16.143085 1278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:29:16.143260 1263 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:29:16.143373 1263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/minicluster-data/ts-0/wal
uuid: "a4a6e4cf9f9d402db560b2202466ba1c"
format_stamp: "Formatted at 2026-09-28 06:29:16 on dist-test-slave-d59m"
server_key: "2753a2cbd13523c3ea40f54debdcb6ef"
server_key_iv: "a817370ce9a91dbacab290b67e21035a"
server_key_version: "encryptionkey@0"
I20260928 06:29:16.143465 1263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:29:16.158936 1263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:29:16.161170 1263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:29:16.161346 1263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:29:16.161617 1263 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:29:16.161938 1263 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:29:16.161996 1263 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:29:16.162038 1263 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:29:16.162060 1263 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:29:16.168424 1263 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.129:36091
I20260928 06:29:16.168466 1391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.129:36091 every 8 connection(s)
I20260928 06:29:16.168836 1263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
I20260928 06:29:16.174356 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 1263
I20260928 06:29:16.174461 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/minicluster-data/ts-0/wal/instance
I20260928 06:29:16.174678 23430 external_mini_cluster.cc:1428] Setting key 0d7988e1fb1f09e9c06adf67c1f69cc5
Sep 28 06:29:16 dist-test-slave-d59m krb5kdc[1156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576956, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.129@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:29:16.179082 1241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:16.169615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:34291 (local address 127.22.225.190:33009)
0928 06:29:16.169686 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 06:29:16.169688 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:29:16.169872 (+ 184us) server_negotiation.cc:407] Connection header received
0928 06:29:16.170406 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:29:16.170407 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:29:16.170420 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:29:16.170480 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:29:16.171423 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:16.171943 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:16.173029 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:16.173243 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:16.173338 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:16.175893 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:29:16.175894 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:29:16.175895 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:29:16.175903 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:29:16.177541 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:29:16.177970 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:29:16.177971 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:29:16.177971 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:29:16.178036 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:29:16.178332 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:29:16.178333 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:29:16.178333 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:29:16.178473 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:29:16.178546 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 06:29:16.178894 (+ 348us) server_negotiation.cc:299] Negotiation successful
0928 06:29:16.178996 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 06:29:16.179306 1394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:16.169575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:33009 (local address 127.22.225.129:34291)
0928 06:29:16.169735 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 06:29:16.169782 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 06:29:16.170292 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:29:16.170555 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:29:16.170559 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:29:16.171122 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:29:16.171311 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:16.171315 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:16.172050 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:16.172051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:16.172938 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:16.172940 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:16.173347 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:16.173348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:16.173474 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:16.174026 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:29:16.174035 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:29:16.175783 (+ 1748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:29:16.177646 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:29:16.177653 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:29:16.177655 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:29:16.177872 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:29:16.178111 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:29:16.178112 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:29:16.178112 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:29:16.178249 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:29:16.178558 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:29:16.178565 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:29:16.178734 (+ 169us) client_negotiation.cc:769] Sending connection context
0928 06:29:16.178851 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 06:29:16.179024 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":83,"threads_started":1}
I20260928 06:29:16.180074 1392 heartbeater.cc:347] Connected to a master server at 127.22.225.190:33009
I20260928 06:29:16.180182 1392 heartbeater.cc:464] Registering TS with master...
I20260928 06:29:16.180385 1392 heartbeater.cc:511] Master 127.22.225.190:33009 requested a full tablet report, sending...
I20260928 06:29:16.181027 1201 ts_manager.cc:194] Registered new tserver with Master: a4a6e4cf9f9d402db560b2202466ba1c (127.22.225.129:36091)
I20260928 06:29:16.182017 1201 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.225.129@KRBTEST.COM'} at 127.22.225.129:34291
WARNING: no policy specified for kudu/127.22.225.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab.
Entry for principal kudu/127.22.225.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab.
I20260928 06:29:16.204499 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.130:0
--local_ip_for_outbound_sockets=127.22.225.130
--webserver_interface=127.22.225.130
--webserver_port=0
--tserver_master_addrs=127.22.225.190:33009
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab
--principal=kudu/127.22.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:41577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:29:16.287808 1397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:29:16.288015 1397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:29:16.288038 1397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:29:16.289319 1397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:29:16.289359 1397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:29:16.289386 1397 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.130
I20260928 06:29:16.291445 1397 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:41577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab
--principal=kudu/127.22.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.225.130
--webserver_port=0
--tserver_master_addrs=127.22.225.190:33009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.130
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:29:16.291719 1397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:29:16.291983 1397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:29:16.297767 1403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:29:16.298007 1405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:29:16.298218 1402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:29:16.298288 1397 server_base.cc:1034] running on GCE node
I20260928 06:29:16.298519 1397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:29:16.298838 1397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:29:16.300012 1397 hybrid_clock.cc:648] HybridClock initialized: now 1790576956299995 us; error 32 us; skew 500 ppm
Sep 28 06:29:16 dist-test-slave-d59m krb5kdc[1156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576956, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:29:16.303267 1397 init.cc:377] Logged in from keytab as kudu/127.22.225.130@KRBTEST.COM (short username kudu)
I20260928 06:29:16.303815 1397 webserver.cc:460] Webserver started at http://127.22.225.130:34029/ using document root <none> and password file <none>
I20260928 06:29:16.304105 1397 fs_manager.cc:359] Metadata directory not provided
I20260928 06:29:16.304177 1397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:29:16.304329 1397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:29:16.305943 1397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/minicluster-data/ts-1/data/instance:
uuid: "b4aa5845b8ce4e48b3b950b41efa4898"
format_stamp: "Formatted at 2026-09-28 06:29:16 on dist-test-slave-d59m"
server_key: "1c9483d76fa37e945081c2e4a11c320d"
server_key_iv: "ad5b47b5164e7bc41fa8c0e6d5c614de"
server_key_version: "encryptionkey@0"
I20260928 06:29:16.306375 1397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/minicluster-data/ts-1/wal/instance:
uuid: "b4aa5845b8ce4e48b3b950b41efa4898"
format_stamp: "Formatted at 2026-09-28 06:29:16 on dist-test-slave-d59m"
server_key: "1c9483d76fa37e945081c2e4a11c320d"
server_key_iv: "ad5b47b5164e7bc41fa8c0e6d5c614de"
server_key_version: "encryptionkey@0"
I20260928 06:29:16.308173 1397 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:29:16.309108 1412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:29:16.309288 1397 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:29:16.309408 1397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/minicluster-data/ts-1/wal
uuid: "b4aa5845b8ce4e48b3b950b41efa4898"
format_stamp: "Formatted at 2026-09-28 06:29:16 on dist-test-slave-d59m"
server_key: "1c9483d76fa37e945081c2e4a11c320d"
server_key_iv: "ad5b47b5164e7bc41fa8c0e6d5c614de"
server_key_version: "encryptionkey@0"
I20260928 06:29:16.309485 1397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:29:16.343245 1397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:29:16.345641 1397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:29:16.345839 1397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:29:16.346118 1397 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:29:16.346501 1397 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:29:16.346560 1397 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:29:16.346604 1397 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:29:16.346630 1397 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:29:16.353014 1397 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.130:40107
I20260928 06:29:16.353091 1525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.130:40107 every 8 connection(s)
I20260928 06:29:16.353415 1397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
I20260928 06:29:16.359736 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 1397
I20260928 06:29:16.359858 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/minicluster-data/ts-1/wal/instance
I20260928 06:29:16.360044 23430 external_mini_cluster.cc:1428] Setting key 36bea9fd458954be7aabe8ce8b361827
Sep 28 06:29:16 dist-test-slave-d59m krb5kdc[1156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576956, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.130@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:29:16.364312 1241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:16.354161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:38339 (local address 127.22.225.190:33009)
0928 06:29:16.354213 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 06:29:16.354215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:29:16.354445 (+ 230us) server_negotiation.cc:407] Connection header received
0928 06:29:16.355135 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:29:16.355136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:29:16.355146 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:29:16.355200 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:29:16.356419 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:16.356990 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:16.358333 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:16.358506 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:16.358555 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:16.361115 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:29:16.361116 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:29:16.361116 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:29:16.361124 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:29:16.362687 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:29:16.363132 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:29:16.363133 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:29:16.363133 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:29:16.363198 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:29:16.363530 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:29:16.363531 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:29:16.363531 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:29:16.363678 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:29:16.363758 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 06:29:16.364112 (+ 354us) server_negotiation.cc:299] Negotiation successful
0928 06:29:16.364207 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 06:29:16.364807 1528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:16.354136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:33009 (local address 127.22.225.130:38339)
0928 06:29:16.354305 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 06:29:16.354355 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 06:29:16.355019 (+ 664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:29:16.355280 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:29:16.355291 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:29:16.356042 (+ 751us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:29:16.356328 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:16.356335 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:16.357088 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:16.357089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:16.358246 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:16.358248 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:16.358588 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:16.358589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:16.358724 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:16.359251 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:29:16.359261 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:29:16.361000 (+ 1739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:29:16.362802 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:29:16.362809 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:29:16.362811 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:29:16.363035 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:29:16.363271 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:29:16.363272 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:29:16.363272 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:29:16.363388 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:29:16.363765 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:29:16.363772 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:29:16.363947 (+ 175us) client_negotiation.cc:769] Sending connection context
0928 06:29:16.364066 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 06:29:16.364242 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":87,"threads_started":1}
I20260928 06:29:16.365334 1526 heartbeater.cc:347] Connected to a master server at 127.22.225.190:33009
I20260928 06:29:16.365434 1526 heartbeater.cc:464] Registering TS with master...
I20260928 06:29:16.365629 1526 heartbeater.cc:511] Master 127.22.225.190:33009 requested a full tablet report, sending...
I20260928 06:29:16.366133 1201 ts_manager.cc:194] Registered new tserver with Master: b4aa5845b8ce4e48b3b950b41efa4898 (127.22.225.130:40107)
I20260928 06:29:16.366518 1201 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.225.130@KRBTEST.COM'} at 127.22.225.130:38339
WARNING: no policy specified for kudu/127.22.225.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab.
Entry for principal kudu/127.22.225.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab.
I20260928 06:29:16.387312 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.131:0
--local_ip_for_outbound_sockets=127.22.225.131
--webserver_interface=127.22.225.131
--webserver_port=0
--tserver_master_addrs=127.22.225.190:33009
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab
--principal=kudu/127.22.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:41577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:29:16.473394 1531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:29:16.473659 1531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:29:16.473701 1531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:29:16.475045 1531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:29:16.475119 1531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:29:16.475167 1531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.131
I20260928 06:29:16.476722 1531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:41577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab
--principal=kudu/127.22.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.225.131
--webserver_port=0
--tserver_master_addrs=127.22.225.190:33009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.131
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:29:16.476992 1531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:29:16.477217 1531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:29:16.482185 1539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:29:16.482218 1537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:29:16.482367 1531 server_base.cc:1034] running on GCE node
W20260928 06:29:16.482442 1536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:29:16.482633 1531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:29:16.482911 1531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:29:16.484079 1531 hybrid_clock.cc:648] HybridClock initialized: now 1790576956484060 us; error 33 us; skew 500 ppm
Sep 28 06:29:16 dist-test-slave-d59m krb5kdc[1156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576956, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:29:16.486617 1531 init.cc:377] Logged in from keytab as kudu/127.22.225.131@KRBTEST.COM (short username kudu)
I20260928 06:29:16.487145 1531 webserver.cc:460] Webserver started at http://127.22.225.131:44221/ using document root <none> and password file <none>
I20260928 06:29:16.487398 1531 fs_manager.cc:359] Metadata directory not provided
I20260928 06:29:16.487443 1531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:29:16.487571 1531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:29:16.488692 1531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/minicluster-data/ts-2/data/instance:
uuid: "b4ffb2b4186c49c0a5ae4327024fda3f"
format_stamp: "Formatted at 2026-09-28 06:29:16 on dist-test-slave-d59m"
server_key: "851001e6132cca6d20b6d609d9e7f430"
server_key_iv: "3f87978bbb3f133f4ff71e08eaecc75f"
server_key_version: "encryptionkey@0"
I20260928 06:29:16.489056 1531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/minicluster-data/ts-2/wal/instance:
uuid: "b4ffb2b4186c49c0a5ae4327024fda3f"
format_stamp: "Formatted at 2026-09-28 06:29:16 on dist-test-slave-d59m"
server_key: "851001e6132cca6d20b6d609d9e7f430"
server_key_iv: "3f87978bbb3f133f4ff71e08eaecc75f"
server_key_version: "encryptionkey@0"
I20260928 06:29:16.490379 1531 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:29:16.491144 1546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:29:16.491298 1531 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:29:16.491406 1531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/minicluster-data/ts-2/wal
uuid: "b4ffb2b4186c49c0a5ae4327024fda3f"
format_stamp: "Formatted at 2026-09-28 06:29:16 on dist-test-slave-d59m"
server_key: "851001e6132cca6d20b6d609d9e7f430"
server_key_iv: "3f87978bbb3f133f4ff71e08eaecc75f"
server_key_version: "encryptionkey@0"
I20260928 06:29:16.491477 1531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:29:16.514022 1531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:29:16.516729 1531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:29:16.516896 1531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:29:16.517179 1531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:29:16.517535 1531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:29:16.517570 1531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:29:16.517666 1531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:29:16.517694 1531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:29:16.525053 1531 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.131:43451
I20260928 06:29:16.525082 1659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.131:43451 every 8 connection(s)
I20260928 06:29:16.525425 1531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
Sep 28 06:29:16 dist-test-slave-d59m krb5kdc[1156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576956, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.131@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:29:16.532521 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 1531
I20260928 06:29:16.532639 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/minicluster-data/ts-2/wal/instance
I20260928 06:29:16.532825 23430 external_mini_cluster.cc:1428] Setting key af3a2bcc3906e0470a9cfc23f3cdde1a
I20260928 06:29:16.535645 1241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:16.526258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:53511 (local address 127.22.225.190:33009)
0928 06:29:16.526309 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 06:29:16.526311 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:29:16.526527 (+ 216us) server_negotiation.cc:407] Connection header received
0928 06:29:16.527088 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:29:16.527089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:29:16.527103 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:29:16.527155 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:29:16.528140 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:16.528746 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:16.529777 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:16.529989 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:16.530101 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:16.532343 (+ 2242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:29:16.532344 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:29:16.532344 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:29:16.532352 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:29:16.534181 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:29:16.534605 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:29:16.534606 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:29:16.534607 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:29:16.534667 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:29:16.534961 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:29:16.534962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:29:16.534963 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:29:16.535091 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:29:16.535171 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 06:29:16.535467 (+ 296us) server_negotiation.cc:299] Negotiation successful
0928 06:29:16.535548 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:29:16.535912 1662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:16.526221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:33009 (local address 127.22.225.131:53511)
0928 06:29:16.526396 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 06:29:16.526435 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 06:29:16.526967 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:29:16.527226 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:29:16.527231 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:29:16.527831 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:29:16.528029 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:16.528034 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:16.528852 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:16.528853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:16.529662 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:16.529664 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:16.530115 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:16.530115 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:16.530244 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:16.530832 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:29:16.530841 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:29:16.532203 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:29:16.534275 (+ 2072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:29:16.534282 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:29:16.534284 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:29:16.534520 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:29:16.534762 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:29:16.534763 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:29:16.534763 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:29:16.534881 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:29:16.535176 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:29:16.535183 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:29:16.535329 (+ 146us) client_negotiation.cc:769] Sending connection context
0928 06:29:16.535444 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 06:29:16.535610 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":101,"threads_started":1}
I20260928 06:29:16.536564 1660 heartbeater.cc:347] Connected to a master server at 127.22.225.190:33009
I20260928 06:29:16.536737 1660 heartbeater.cc:464] Registering TS with master...
I20260928 06:29:16.536938 1660 heartbeater.cc:511] Master 127.22.225.190:33009 requested a full tablet report, sending...
I20260928 06:29:16.537345 1201 ts_manager.cc:194] Registered new tserver with Master: b4ffb2b4186c49c0a5ae4327024fda3f (127.22.225.131:43451)
I20260928 06:29:16.537748 1201 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.225.131@KRBTEST.COM'} at 127.22.225.131:53511
I20260928 06:29:16.544804 23430 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:29:16.552635 1241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:16.546186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59862 (local address 127.22.225.190:33009)
0928 06:29:16.546247 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 06:29:16.546249 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:29:16.546380 (+ 131us) server_negotiation.cc:407] Connection header received
0928 06:29:16.546444 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:29:16.546445 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:29:16.546457 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:29:16.546507 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:29:16.547117 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:16.547771 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:16.548479 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:16.548739 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:16.548809 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:16.550025 (+ 1216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:29:16.550026 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:29:16.550026 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:29:16.550034 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:29:16.551475 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:29:16.551796 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:29:16.551797 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:29:16.551797 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:29:16.551862 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:29:16.552047 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:29:16.552048 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:29:16.552048 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:29:16.552238 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:29:16.552358 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 06:29:16.552447 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 06:29:16.552524 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 06:29:16.554461 1201 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59862:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:29:16.556005 1201 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:29:16.562871 1672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:16.558424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.130:40107 (local address 127.0.0.1:37286)
0928 06:29:16.558695 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0928 06:29:16.558705 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:29:16.558765 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:29:16.558994 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:29:16.558997 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:29:16.558998 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:29:16.559063 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:16.559064 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:16.560203 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:16.560204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:16.561580 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:16.561583 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:16.562583 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:16.562584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:16.562669 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:16.562680 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 06:29:16.562736 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 06:29:16.562781 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":104,"threads_started":1}
I20260928 06:29:16.563206 1671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:16.558419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.131:43451 (local address 127.0.0.1:45642)
0928 06:29:16.558535 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 06:29:16.558575 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 06:29:16.558639 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:29:16.559419 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:29:16.559420 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:29:16.559422 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:29:16.559480 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:16.559482 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:16.560675 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:16.560677 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:16.561947 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:16.561950 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:16.562868 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:16.562869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:16.563027 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:16.563031 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:29:16.563077 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 06:29:16.563112 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"mutex_wait_us":2,"thread_start_us":66,"threads_started":1}
I20260928 06:29:16.563208 1673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:16.558538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37286 (local address 127.22.225.130:40107)
0928 06:29:16.558676 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 06:29:16.558682 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:29:16.558828 (+ 146us) server_negotiation.cc:407] Connection header received
0928 06:29:16.558872 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:29:16.558875 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:29:16.558891 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:29:16.558974 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:29:16.559131 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:16.560100 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:16.561682 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:16.562410 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:16.562459 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:16.562522 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 06:29:16.563057 (+ 535us) server_negotiation.cc:299] Negotiation successful
0928 06:29:16.563099 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":73,"threads_started":1}
I20260928 06:29:16.563794 1675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:16.558938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45642 (local address 127.22.225.131:43451)
0928 06:29:16.559198 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0928 06:29:16.559203 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:29:16.559213 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:29:16.559250 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:29:16.559252 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:29:16.559268 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:29:16.559342 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:29:16.559546 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:16.560511 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:16.562039 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:16.562712 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:16.562785 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:16.562834 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 06:29:16.563671 (+ 837us) server_negotiation.cc:299] Negotiation successful
0928 06:29:16.563715 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":83,"threads_started":1}
I20260928 06:29:16.563966 1460 tablet_service.cc:1467] Processing CreateTablet for tablet 8b1538a163df43deaf3cbac3a4a7c2ee (DEFAULT_TABLE table=test-table [id=3e01dfbf740e4f3583141317866899ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:29:16.564253 1674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:16.558654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.129:36091 (local address 127.0.0.1:45982)
0928 06:29:16.558768 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 06:29:16.558777 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:29:16.558821 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:29:16.560460 (+ 1639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:29:16.560460 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:29:16.560462 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:29:16.560495 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:16.560497 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:16.561786 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:16.561787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:16.562818 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:16.562820 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:16.564031 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:16.564032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:16.564089 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:16.564093 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:29:16.564154 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 06:29:16.564190 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":80,"threads_started":1}
I20260928 06:29:16.564313 1676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:16.559111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45982 (local address 127.22.225.129:36091)
0928 06:29:16.560296 (+ 1185us) server_negotiation.cc:206] Beginning negotiation
0928 06:29:16.560300 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:29:16.560309 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:29:16.560345 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:29:16.560347 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:29:16.560363 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:29:16.560572 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:29:16.560665 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:16.561677 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:16.562912 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:16.563818 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:16.564031 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:16.564091 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 06:29:16.564167 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 06:29:16.564201 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1141,"thread_start_us":66,"threads_started":1}
I20260928 06:29:16.564366 1460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b1538a163df43deaf3cbac3a4a7c2ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:29:16.564747 1594 tablet_service.cc:1467] Processing CreateTablet for tablet 8b1538a163df43deaf3cbac3a4a7c2ee (DEFAULT_TABLE table=test-table [id=3e01dfbf740e4f3583141317866899ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:29:16.565049 1594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b1538a163df43deaf3cbac3a4a7c2ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:29:16.567512 1677 tablet_bootstrap.cc:492] T 8b1538a163df43deaf3cbac3a4a7c2ee P b4ffb2b4186c49c0a5ae4327024fda3f: Bootstrap starting.
I20260928 06:29:16.568079 1678 tablet_bootstrap.cc:492] T 8b1538a163df43deaf3cbac3a4a7c2ee P b4aa5845b8ce4e48b3b950b41efa4898: Bootstrap starting.
I20260928 06:29:16.568325 1677 tablet_bootstrap.cc:654] T 8b1538a163df43deaf3cbac3a4a7c2ee P b4ffb2b4186c49c0a5ae4327024fda3f: Neither blocks nor log segments found. Creating new log.
I20260928 06:29:16.568408 1326 tablet_service.cc:1467] Processing CreateTablet for tablet 8b1538a163df43deaf3cbac3a4a7c2ee (DEFAULT_TABLE table=test-table [id=3e01dfbf740e4f3583141317866899ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:29:16.568773 1326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b1538a163df43deaf3cbac3a4a7c2ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:29:16.568789 1678 tablet_bootstrap.cc:654] T 8b1538a163df43deaf3cbac3a4a7c2ee P b4aa5845b8ce4e48b3b950b41efa4898: Neither blocks nor log segments found. Creating new log.
I20260928 06:29:16.569243 1677 log.cc:824] T 8b1538a163df43deaf3cbac3a4a7c2ee P b4ffb2b4186c49c0a5ae4327024fda3f: Log is configured to *not* fsync() on all Append() calls
I20260928 06:29:16.569351 1678 log.cc:824] T 8b1538a163df43deaf3cbac3a4a7c2ee P b4aa5845b8ce4e48b3b950b41efa4898: Log is configured to *not* fsync() on all Append() calls
I20260928 06:29:16.570065 1678 tablet_bootstrap.cc:492] T 8b1538a163df43deaf3cbac3a4a7c2ee P b4aa5845b8ce4e48b3b950b41efa4898: No bootstrap required, opened a new log
I20260928 06:29:16.570071 1677 tablet_bootstrap.cc:492] T 8b1538a163df43deaf3cbac3a4a7c2ee P b4ffb2b4186c49c0a5ae4327024fda3f: No bootstrap required, opened a new log
I20260928 06:29:16.570132 1677 ts_tablet_manager.cc:1397] T 8b1538a163df43deaf3cbac3a4a7c2ee P b4ffb2b4186c49c0a5ae4327024fda3f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 06:29:16.570132 1678 ts_tablet_manager.cc:1397] T 8b1538a163df43deaf3cbac3a4a7c2ee P b4aa5845b8ce4e48b3b950b41efa4898: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:29:16.572055 1677 raft_consensus.cc:348] T 8b1538a163df43deaf3cbac3a4a7c2ee P b4ffb2b4186c49c0a5ae4327024fda3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4ffb2b4186c49c0a5ae4327024fda3f" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 43451 } } peers { permanent_uuid: "b4aa5845b8ce4e48b3b950b41efa4898" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 40107 } } peers { permanent_uuid: "a4a6e4cf9f9d402db560b2202466ba1c" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 36091 } }
I20260928 06:29:16.572217 1677 raft_consensus.cc:374] T 8b1538a163df43deaf3cbac3a4a7c2ee P b4ffb2b4186c49c0a5ae4327024fda3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:29:16.572260 1677 raft_consensus.cc:729] T 8b1538a163df43deaf3cbac3a4a7c2ee P b4ffb2b4186c49c0a5ae4327024fda3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4ffb2b4186c49c0a5ae4327024fda3f, State: Initialized, Role: FOLLOWER
I20260928 06:29:16.572176 1678 raft_consensus.cc:348] T 8b1538a163df43deaf3cbac3a4a7c2ee P b4aa5845b8ce4e48b3b950b41efa4898 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4ffb2b4186c49c0a5ae4327024fda3f" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 43451 } } peers { permanent_uuid: "b4aa5845b8ce4e48b3b950b41efa4898" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 40107 } } peers { permanent_uuid: "a4a6e4cf9f9d402db560b2202466ba1c" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 36091 } }
I20260928 06:29:16.572300 1678 raft_consensus.cc:374] T 8b1538a163df43deaf3cbac3a4a7c2ee P b4aa5845b8ce4e48b3b950b41efa4898 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:29:16.572328 1678 raft_consensus.cc:729] T 8b1538a163df43deaf3cbac3a4a7c2ee P b4aa5845b8ce4e48b3b950b41efa4898 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4aa5845b8ce4e48b3b950b41efa4898, State: Initialized, Role: FOLLOWER
I20260928 06:29:16.572372 1677 consensus_queue.cc:260] T 8b1538a163df43deaf3cbac3a4a7c2ee P b4ffb2b4186c49c0a5ae4327024fda3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4ffb2b4186c49c0a5ae4327024fda3f" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 43451 } } peers { permanent_uuid: "b4aa5845b8ce4e48b3b950b41efa4898" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 40107 } } peers { permanent_uuid: "a4a6e4cf9f9d402db560b2202466ba1c" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 36091 } }
I20260928 06:29:16.572420 1678 consensus_queue.cc:260] T 8b1538a163df43deaf3cbac3a4a7c2ee P b4aa5845b8ce4e48b3b950b41efa4898 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4ffb2b4186c49c0a5ae4327024fda3f" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 43451 } } peers { permanent_uuid: "b4aa5845b8ce4e48b3b950b41efa4898" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 40107 } } peers { permanent_uuid: "a4a6e4cf9f9d402db560b2202466ba1c" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 36091 } }
I20260928 06:29:16.572710 1677 ts_tablet_manager.cc:1428] T 8b1538a163df43deaf3cbac3a4a7c2ee P b4ffb2b4186c49c0a5ae4327024fda3f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 06:29:16.572710 1678 ts_tablet_manager.cc:1428] T 8b1538a163df43deaf3cbac3a4a7c2ee P b4aa5845b8ce4e48b3b950b41efa4898: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:29:16.572700 1681 tablet_bootstrap.cc:492] T 8b1538a163df43deaf3cbac3a4a7c2ee P a4a6e4cf9f9d402db560b2202466ba1c: Bootstrap starting.
I20260928 06:29:16.572844 1660 heartbeater.cc:503] Master 127.22.225.190:33009 was elected leader, sending a full tablet report...
I20260928 06:29:16.572844 1526 heartbeater.cc:503] Master 127.22.225.190:33009 was elected leader, sending a full tablet report...
I20260928 06:29:16.573390 1681 tablet_bootstrap.cc:654] T 8b1538a163df43deaf3cbac3a4a7c2ee P a4a6e4cf9f9d402db560b2202466ba1c: Neither blocks nor log segments found. Creating new log.
I20260928 06:29:16.574000 1681 log.cc:824] T 8b1538a163df43deaf3cbac3a4a7c2ee P a4a6e4cf9f9d402db560b2202466ba1c: Log is configured to *not* fsync() on all Append() calls
I20260928 06:29:16.574788 1681 tablet_bootstrap.cc:492] T 8b1538a163df43deaf3cbac3a4a7c2ee P a4a6e4cf9f9d402db560b2202466ba1c: No bootstrap required, opened a new log
I20260928 06:29:16.574884 1681 ts_tablet_manager.cc:1397] T 8b1538a163df43deaf3cbac3a4a7c2ee P a4a6e4cf9f9d402db560b2202466ba1c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 06:29:16.576375 1681 raft_consensus.cc:348] T 8b1538a163df43deaf3cbac3a4a7c2ee P a4a6e4cf9f9d402db560b2202466ba1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4ffb2b4186c49c0a5ae4327024fda3f" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 43451 } } peers { permanent_uuid: "b4aa5845b8ce4e48b3b950b41efa4898" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 40107 } } peers { permanent_uuid: "a4a6e4cf9f9d402db560b2202466ba1c" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 36091 } }
I20260928 06:29:16.576552 1681 raft_consensus.cc:374] T 8b1538a163df43deaf3cbac3a4a7c2ee P a4a6e4cf9f9d402db560b2202466ba1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:29:16.576594 1681 raft_consensus.cc:729] T 8b1538a163df43deaf3cbac3a4a7c2ee P a4a6e4cf9f9d402db560b2202466ba1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4a6e4cf9f9d402db560b2202466ba1c, State: Initialized, Role: FOLLOWER
I20260928 06:29:16.576740 1681 consensus_queue.cc:260] T 8b1538a163df43deaf3cbac3a4a7c2ee P a4a6e4cf9f9d402db560b2202466ba1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4ffb2b4186c49c0a5ae4327024fda3f" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 43451 } } peers { permanent_uuid: "b4aa5845b8ce4e48b3b950b41efa4898" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 40107 } } peers { permanent_uuid: "a4a6e4cf9f9d402db560b2202466ba1c" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 36091 } }
I20260928 06:29:16.577019 1681 ts_tablet_manager.cc:1428] T 8b1538a163df43deaf3cbac3a4a7c2ee P a4a6e4cf9f9d402db560b2202466ba1c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 06:29:16.577042 1392 heartbeater.cc:503] Master 127.22.225.190:33009 was elected leader, sending a full tablet report...
W20260928 06:29:16.604056 1527 tablet.cc:2369] T 8b1538a163df43deaf3cbac3a4a7c2ee P b4aa5845b8ce4e48b3b950b41efa4898: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:29:16.670207 1393 tablet.cc:2369] T 8b1538a163df43deaf3cbac3a4a7c2ee P a4a6e4cf9f9d402db560b2202466ba1c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:29:16.754691 1682 raft_consensus.cc:482] T 8b1538a163df43deaf3cbac3a4a7c2ee P b4ffb2b4186c49c0a5ae4327024fda3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:29:16.754846 1682 raft_consensus.cc:504] T 8b1538a163df43deaf3cbac3a4a7c2ee P b4ffb2b4186c49c0a5ae4327024fda3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4ffb2b4186c49c0a5ae4327024fda3f" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 43451 } } peers { permanent_uuid: "b4aa5845b8ce4e48b3b950b41efa4898" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 40107 } } peers { permanent_uuid: "a4a6e4cf9f9d402db560b2202466ba1c" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 36091 } }
I20260928 06:29:16.755247 1682 leader_election.cc:288] T 8b1538a163df43deaf3cbac3a4a7c2ee P b4ffb2b4186c49c0a5ae4327024fda3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b4aa5845b8ce4e48b3b950b41efa4898 (127.22.225.130:40107), a4a6e4cf9f9d402db560b2202466ba1c (127.22.225.129:36091)
I20260928 06:29:16.758756 1673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:16.755736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:58655 (local address 127.22.225.130:40107)
0928 06:29:16.755781 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 06:29:16.755783 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:29:16.755790 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:29:16.755825 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:29:16.755825 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:29:16.755834 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:29:16.755892 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:29:16.756112 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:16.756811 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:16.757789 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:16.758370 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:16.758447 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:16.758455 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:29:16.758648 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 06:29:16.758690 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 06:29:16.758929 1676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:16.755580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:39417 (local address 127.22.225.129:36091)
0928 06:29:16.755624 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 06:29:16.755626 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:29:16.755636 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:29:16.755665 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:29:16.755666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:29:16.755677 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:29:16.755747 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:29:16.755953 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:16.756778 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:16.758000 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:16.758544 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:16.758628 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:16.758636 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:29:16.758818 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 06:29:16.758854 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:29:16.758991 1686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:16.755473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.130:40107 (local address 127.22.225.131:58655)
0928 06:29:16.755673 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 06:29:16.755682 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:29:16.755721 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:29:16.755923 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:29:16.755924 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:29:16.755926 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:29:16.755999 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:16.756001 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:16.756894 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:16.756895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:16.757669 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:16.757670 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:16.758468 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:16.758469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:16.758553 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:16.758557 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:29:16.758853 (+ 296us) client_negotiation.cc:240] Negotiation successful
0928 06:29:16.758889 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":96,"threads_started":1}
I20260928 06:29:16.759214 1662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:16.755456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.129:36091 (local address 127.22.225.131:39417)
0928 06:29:16.755518 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 06:29:16.755531 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:29:16.755577 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:29:16.755792 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:29:16.755793 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:29:16.755795 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:29:16.755877 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:16.755879 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:16.756871 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:16.756872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:16.757890 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:29:16.757892 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:29:16.758628 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:29:16.758629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:29:16.758715 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:16.758717 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:29:16.758810 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 06:29:16.758848 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 06:29:16.759334 1480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1538a163df43deaf3cbac3a4a7c2ee" candidate_uuid: "b4ffb2b4186c49c0a5ae4327024fda3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4aa5845b8ce4e48b3b950b41efa4898" is_pre_election: true
I20260928 06:29:16.759512 1480 raft_consensus.cc:2454] T 8b1538a163df43deaf3cbac3a4a7c2ee P b4aa5845b8ce4e48b3b950b41efa4898 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4ffb2b4186c49c0a5ae4327024fda3f in term 0.
I20260928 06:29:16.759552 1346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1538a163df43deaf3cbac3a4a7c2ee" candidate_uuid: "b4ffb2b4186c49c0a5ae4327024fda3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4a6e4cf9f9d402db560b2202466ba1c" is_pre_election: true
I20260928 06:29:16.759693 1346 raft_consensus.cc:2454] T 8b1538a163df43deaf3cbac3a4a7c2ee P a4a6e4cf9f9d402db560b2202466ba1c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4ffb2b4186c49c0a5ae4327024fda3f in term 0.
I20260928 06:29:16.759774 1549 leader_election.cc:302] T 8b1538a163df43deaf3cbac3a4a7c2ee P b4ffb2b4186c49c0a5ae4327024fda3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4aa5845b8ce4e48b3b950b41efa4898, b4ffb2b4186c49c0a5ae4327024fda3f; no voters:
I20260928 06:29:16.759970 1682 raft_consensus.cc:2781] T 8b1538a163df43deaf3cbac3a4a7c2ee P b4ffb2b4186c49c0a5ae4327024fda3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:29:16.760027 1682 raft_consensus.cc:482] T 8b1538a163df43deaf3cbac3a4a7c2ee P b4ffb2b4186c49c0a5ae4327024fda3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:29:16.760042 1682 raft_consensus.cc:3037] T 8b1538a163df43deaf3cbac3a4a7c2ee P b4ffb2b4186c49c0a5ae4327024fda3f [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:29:16.760876 1682 raft_consensus.cc:504] T 8b1538a163df43deaf3cbac3a4a7c2ee P b4ffb2b4186c49c0a5ae4327024fda3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4ffb2b4186c49c0a5ae4327024fda3f" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 43451 } } peers { permanent_uuid: "b4aa5845b8ce4e48b3b950b41efa4898" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 40107 } } peers { permanent_uuid: "a4a6e4cf9f9d402db560b2202466ba1c" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 36091 } }
I20260928 06:29:16.761023 1682 leader_election.cc:288] T 8b1538a163df43deaf3cbac3a4a7c2ee P b4ffb2b4186c49c0a5ae4327024fda3f [CANDIDATE]: Term 1 election: Requested vote from peers b4aa5845b8ce4e48b3b950b41efa4898 (127.22.225.130:40107), a4a6e4cf9f9d402db560b2202466ba1c (127.22.225.129:36091)
I20260928 06:29:16.761202 1346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1538a163df43deaf3cbac3a4a7c2ee" candidate_uuid: "b4ffb2b4186c49c0a5ae4327024fda3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4a6e4cf9f9d402db560b2202466ba1c"
I20260928 06:29:16.761222 1480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1538a163df43deaf3cbac3a4a7c2ee" candidate_uuid: "b4ffb2b4186c49c0a5ae4327024fda3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4aa5845b8ce4e48b3b950b41efa4898"
I20260928 06:29:16.761305 1480 raft_consensus.cc:3037] T 8b1538a163df43deaf3cbac3a4a7c2ee P b4aa5845b8ce4e48b3b950b41efa4898 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:29:16.761309 1346 raft_consensus.cc:3037] T 8b1538a163df43deaf3cbac3a4a7c2ee P a4a6e4cf9f9d402db560b2202466ba1c [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:29:16.762001 1346 raft_consensus.cc:2454] T 8b1538a163df43deaf3cbac3a4a7c2ee P a4a6e4cf9f9d402db560b2202466ba1c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4ffb2b4186c49c0a5ae4327024fda3f in term 1.
I20260928 06:29:16.762156 1480 raft_consensus.cc:2454] T 8b1538a163df43deaf3cbac3a4a7c2ee P b4aa5845b8ce4e48b3b950b41efa4898 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4ffb2b4186c49c0a5ae4327024fda3f in term 1.
I20260928 06:29:16.762197 1548 leader_election.cc:302] T 8b1538a163df43deaf3cbac3a4a7c2ee P b4ffb2b4186c49c0a5ae4327024fda3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4a6e4cf9f9d402db560b2202466ba1c, b4ffb2b4186c49c0a5ae4327024fda3f; no voters:
I20260928 06:29:16.762322 1682 raft_consensus.cc:2781] T 8b1538a163df43deaf3cbac3a4a7c2ee P b4ffb2b4186c49c0a5ae4327024fda3f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:29:16.762521 1682 raft_consensus.cc:686] T 8b1538a163df43deaf3cbac3a4a7c2ee P b4ffb2b4186c49c0a5ae4327024fda3f [term 1 LEADER]: Becoming Leader. State: Replica: b4ffb2b4186c49c0a5ae4327024fda3f, State: Running, Role: LEADER
I20260928 06:29:16.762619 1682 consensus_queue.cc:237] T 8b1538a163df43deaf3cbac3a4a7c2ee P b4ffb2b4186c49c0a5ae4327024fda3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4ffb2b4186c49c0a5ae4327024fda3f" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 43451 } } peers { permanent_uuid: "b4aa5845b8ce4e48b3b950b41efa4898" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 40107 } } peers { permanent_uuid: "a4a6e4cf9f9d402db560b2202466ba1c" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 36091 } }
I20260928 06:29:16.763594 1201 catalog_manager.cc:5495] T 8b1538a163df43deaf3cbac3a4a7c2ee P b4ffb2b4186c49c0a5ae4327024fda3f reported cstate change: term changed from 0 to 1, leader changed from <none> to b4ffb2b4186c49c0a5ae4327024fda3f (127.22.225.131). New cstate: current_term: 1 leader_uuid: "b4ffb2b4186c49c0a5ae4327024fda3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4ffb2b4186c49c0a5ae4327024fda3f" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 43451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4aa5845b8ce4e48b3b950b41efa4898" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 40107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4a6e4cf9f9d402db560b2202466ba1c" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 36091 } health_report { overall_health: UNKNOWN } } }
W20260928 06:29:16.776033 1661 tablet.cc:2369] T 8b1538a163df43deaf3cbac3a4a7c2ee P b4ffb2b4186c49c0a5ae4327024fda3f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:29:16.790951 1675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:16.788084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45654 (local address 127.22.225.131:43451)
0928 06:29:16.788134 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 06:29:16.788136 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:29:16.788147 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:29:16.788205 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:29:16.788206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:29:16.788233 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:29:16.788317 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:29:16.788471 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:16.788952 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:16.789866 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:29:16.790203 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:29:16.790269 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:29:16.790508 (+ 239us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:29:16.790698 (+ 190us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:29:16.790788 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 06:29:16.790815 (+ 27us) server_negotiation.cc:299] Negotiation successful
0928 06:29:16.790836 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 06:29:16.792802 1480 raft_consensus.cc:1261] T 8b1538a163df43deaf3cbac3a4a7c2ee P b4aa5845b8ce4e48b3b950b41efa4898 [term 1 FOLLOWER]: Refusing update from remote peer b4ffb2b4186c49c0a5ae4327024fda3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:29:16.792802 1346 raft_consensus.cc:1261] T 8b1538a163df43deaf3cbac3a4a7c2ee P a4a6e4cf9f9d402db560b2202466ba1c [term 1 FOLLOWER]: Refusing update from remote peer b4ffb2b4186c49c0a5ae4327024fda3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:29:16.793179 1682 consensus_queue.cc:1035] T 8b1538a163df43deaf3cbac3a4a7c2ee P b4ffb2b4186c49c0a5ae4327024fda3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4a6e4cf9f9d402db560b2202466ba1c" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 36091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:29:16.793340 1687 consensus_queue.cc:1035] T 8b1538a163df43deaf3cbac3a4a7c2ee P b4ffb2b4186c49c0a5ae4327024fda3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4aa5845b8ce4e48b3b950b41efa4898" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 40107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:29:16.796015 1691 mvcc.cc:205] Tried to move back new op lower bound from 7334203215020355584 to 7334203214900289536. Current Snapshot: MvccSnapshot[applied={T|T < 7334203215020355584}]
I20260928 06:29:16.797717 1690 mvcc.cc:205] Tried to move back new op lower bound from 7334203215020355584 to 7334203214900289536. Current Snapshot: MvccSnapshot[applied={T|T < 7334203215020355584 or (T in {7334203215020355584})}]
I20260928 06:29:16.798195 1693 mvcc.cc:205] Tried to move back new op lower bound from 7334203215020355584 to 7334203214900289536. Current Snapshot: MvccSnapshot[applied={T|T < 7334203215020355584}]
I20260928 06:29:16.798743 1201 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:29:16.798858 1201 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:29:16.799635 1201 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a42ae672dbb847cabc5a75452074b9a3: Sending DeleteTablet for 3 replicas of tablet 8b1538a163df43deaf3cbac3a4a7c2ee
I20260928 06:29:16.800118 1326 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b1538a163df43deaf3cbac3a4a7c2ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:29:16 UTC) from {username='kudu', principal='kudu/127.22.225.190@KRBTEST.COM'} at 127.0.0.1:45982
I20260928 06:29:16.800230 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 1263
I20260928 06:29:16.800330 1460 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b1538a163df43deaf3cbac3a4a7c2ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:29:16 UTC) from {username='kudu', principal='kudu/127.22.225.190@KRBTEST.COM'} at 127.0.0.1:37286
I20260928 06:29:16.801155 1594 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b1538a163df43deaf3cbac3a4a7c2ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:29:16 UTC) from {username='kudu', principal='kudu/127.22.225.190@KRBTEST.COM'} at 127.0.0.1:45642
I20260928 06:29:16.802006 1701 tablet_replica.cc:330] T 8b1538a163df43deaf3cbac3a4a7c2ee P b4ffb2b4186c49c0a5ae4327024fda3f: stopping tablet replica
I20260928 06:29:16.802104 1701 raft_consensus.cc:2229] T 8b1538a163df43deaf3cbac3a4a7c2ee P b4ffb2b4186c49c0a5ae4327024fda3f [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:29:16.802198 1701 raft_consensus.cc:2258] T 8b1538a163df43deaf3cbac3a4a7c2ee P b4ffb2b4186c49c0a5ae4327024fda3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:29:16.802585 1701 ts_tablet_manager.cc:1905] T 8b1538a163df43deaf3cbac3a4a7c2ee P b4ffb2b4186c49c0a5ae4327024fda3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:29:16.804279 1701 ts_tablet_manager.cc:1918] T 8b1538a163df43deaf3cbac3a4a7c2ee P b4ffb2b4186c49c0a5ae4327024fda3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:29:16.804409 1701 log.cc:1196] T 8b1538a163df43deaf3cbac3a4a7c2ee P b4ffb2b4186c49c0a5ae4327024fda3f: Deleting WAL directory at /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576899143097-23430-0/minicluster-data/ts-2/wal/wals/8b1538a163df43deaf3cbac3a4a7c2ee
I20260928 06:29:16.804780 1701 ts_tablet_manager.cc:1939] T 8b1538a163df43deaf3cbac3a4a7c2ee P b4ffb2b4186c49c0a5ae4327024fda3f: Deleting consensus metadata
I20260928 06:29:16.805236 1186 catalog_manager.cc:4841] TS b4ffb2b4186c49c0a5ae4327024fda3f (127.22.225.131:43451): tablet 8b1538a163df43deaf3cbac3a4a7c2ee (table test-table [id=3e01dfbf740e4f3583141317866899ca]) successfully deleted
W20260928 06:29:16.806969 1186 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.225.129:36091 (error 108)
I20260928 06:29:16.807112 1700 tablet_replica.cc:330] T 8b1538a163df43deaf3cbac3a4a7c2ee P b4aa5845b8ce4e48b3b950b41efa4898: stopping tablet replica
I20260928 06:29:16.807217 1700 raft_consensus.cc:2229] T 8b1538a163df43deaf3cbac3a4a7c2ee P b4aa5845b8ce4e48b3b950b41efa4898 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:29:16.807329 1700 raft_consensus.cc:2258] T 8b1538a163df43deaf3cbac3a4a7c2ee P b4aa5845b8ce4e48b3b950b41efa4898 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:29:16.807495 1674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:16.807335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.129:36091 (local address 127.0.0.1:45984)
0928 06:29:16.807412 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0928 06:29:16.807458 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.129:36091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 06:29:16.807574 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 1397
W20260928 06:29:16.807605 1186 catalog_manager.cc:4584] TS a4a6e4cf9f9d402db560b2202466ba1c (127.22.225.129:36091): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8b1538a163df43deaf3cbac3a4a7c2ee: Network error: Client connection negotiation failed: client connection to 127.22.225.129:36091: connect: Connection refused (error 111)
I20260928 06:29:16.815951 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 1531
I20260928 06:29:16.815968 1674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:29:16.815852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.130:40107 (local address 127.0.0.1:37296)
0928 06:29:16.815901 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 06:29:16.815928 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.130:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 06:29:16.820906 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 1169
2026-09-28T06:29:16Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2263 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:29:16 dist-test-slave-d59m krb5kdc[1705](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:29:16 dist-test-slave-d59m krb5kdc[1705](info): set up 2 sockets
Sep 28 06:29:16 dist-test-slave-d59m krb5kdc[1705](info): commencing operation
krb5kdc: starting...
W20260928 06:29:17.898236 23430 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Sep 28 06:29:17 dist-test-slave-d59m krb5kdc[1705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:29:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:29:17Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.22.225.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.22.225.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.22.225.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/krb5kdc/rangeradmin_127.22.225.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.22.225.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/krb5kdc/rangeradmin_127.22.225.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.22.225.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.22.225.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.22.225.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/krb5kdc/rangerlookup_127.22.225.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.22.225.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/krb5kdc/rangerlookup_127.22.225.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.22.225.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.225.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.225.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/krb5kdc/HTTP_127.22.225.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.22.225.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/krb5kdc/HTTP_127.22.225.148@KRBTEST.COM.keytab.
I20260928 06:29:18.069170 23430 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskcCVV2M/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/postgres -l logfile start
2026-09-28 06:29:20.785 UTC [1735] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 06:29:20.785 UTC [1735] LOG: listening on IPv4 address "127.22.225.148", port 48769
2026-09-28 06:29:20.788 UTC [1735] LOG: listening on Unix socket "/tmp/.s.PGSQL.48769"
2026-09-28 06:29:20.812 UTC [1738] LOG: database system was shut down at 2026-09-28 06:29:19 UTC
2026-09-28 06:29:20.816 UTC [1735] LOG: database system is ready to accept connections
I20260928 06:29:21.782066 23430 mini_postgres.cc:96] Postgres bound to 48769
2026-09-28 06:29:21.789 UTC [1746] FATAL: database "slave" does not exist
127.22.225.148:48769 - accepting connections
I20260928 06:29:21.790032 23430 mini_ranger.cc:161] Starting Ranger...
I20260928 06:29:21.811295 23430 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 06:29:22.356719 23430 mini_ranger.cc:87] Created ranger Postgres database
I20260928 06:29:22.356813 23430 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/ranger-admin
2026-09-28 06:29:22,832 [I] DB FLAVOR :POSTGRES
2026-09-28 06:29:22,832 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 06:29:22,833 [I] Checking connection..
2026-09-28 06:29:22,833 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcCVV2M/build/release/bin/postgresql.jar:/tmp/dist-test-taskcCVV2M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.225.148:48769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 06:29:23,178 [I] Checking connection passed.
2026-09-28 06:29:23,179 [I] --------- Verifying version history table ---------
2026-09-28 06:29:23,179 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcCVV2M/build/release/bin/postgresql.jar:/tmp/dist-test-taskcCVV2M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.225.148:48769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 06:29:23,485 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 06:29:23,485 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcCVV2M/build/release/bin/postgresql.jar:/tmp/dist-test-taskcCVV2M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.225.148:48769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 06:29:23,771 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 06:29:23,771 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 06:29:23,771 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcCVV2M/build/release/bin/postgresql.jar:/tmp/dist-test-taskcCVV2M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.225.148:48769/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcCVV2M/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 06:29:24.066 UTC [1810] WARNING: there is no transaction in progress
2026-09-28 06:29:24,073 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 06:29:24,074 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcCVV2M/build/release/bin/postgresql.jar:/tmp/dist-test-taskcCVV2M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.225.148:48769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 06:29:24,364 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 06:29:24,365 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 06:29:24,365 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcCVV2M/build/release/bin/postgresql.jar:/tmp/dist-test-taskcCVV2M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.225.148:48769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 06:29:24,685 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcCVV2M/build/release/bin/postgresql.jar:/tmp/dist-test-taskcCVV2M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.225.148:48769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 06:29:24,967 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcCVV2M/build/release/bin/postgresql.jar:/tmp/dist-test-taskcCVV2M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.225.148:48769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-d59m.c.gcp-upstream.internal','N') ;"
2026-09-28 06:29:25,241 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcCVV2M/build/release/bin/postgresql.jar:/tmp/dist-test-taskcCVV2M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.225.148:48769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 06:29:25,520 [I] Table x_portal_user does not exist in database ranger
2026-09-28 06:29:25,521 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcCVV2M/build/release/bin/postgresql.jar:/tmp/dist-test-taskcCVV2M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.225.148:48769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 06:29:25,800 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 06:29:25,801 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcCVV2M/build/release/bin/postgresql.jar:/tmp/dist-test-taskcCVV2M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.225.148:48769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 06:29:26,107 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcCVV2M/build/release/bin/postgresql.jar:/tmp/dist-test-taskcCVV2M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.225.148:48769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 06:29:26,393 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 06:29:26,394 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcCVV2M/build/release/bin/postgresql.jar:/tmp/dist-test-taskcCVV2M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.225.148:48769/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcCVV2M/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 06:29:27.447 UTC [1918] WARNING: there is no transaction in progress
2026-09-28 06:29:27.465 UTC [1918] WARNING: there is no transaction in progress
2026-09-28 06:29:27.481 UTC [1918] WARNING: there is no transaction in progress
2026-09-28 06:29:27.499 UTC [1918] WARNING: there is no transaction in progress
2026-09-28 06:29:27.517 UTC [1918] WARNING: there is no transaction in progress
2026-09-28 06:29:27.537 UTC [1918] WARNING: there is no transaction in progress
2026-09-28 06:29:27.599 UTC [1918] WARNING: there is no transaction in progress
2026-09-28 06:29:27.606 UTC [1918] WARNING: there is no transaction in progress
2026-09-28 06:29:27.621 UTC [1918] WARNING: there is no transaction in progress
2026-09-28 06:29:27.631 UTC [1918] WARNING: there is no transaction in progress
2026-09-28 06:29:27.640 UTC [1918] WARNING: there is no transaction in progress
2026-09-28 06:29:27.656 UTC [1918] WARNING: there is no transaction in progress
2026-09-28 06:29:27.668 UTC [1918] WARNING: there is no transaction in progress
2026-09-28 06:29:27.676 UTC [1918] WARNING: there is no transaction in progress
2026-09-28 06:29:27.683 UTC [1918] WARNING: there is no transaction in progress
2026-09-28 06:29:27.690 UTC [1918] WARNING: there is no transaction in progress
2026-09-28 06:29:28,356 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 06:29:28,357 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcCVV2M/build/release/bin/postgresql.jar:/tmp/dist-test-taskcCVV2M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.225.148:48769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 06:29:28,688 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 06:29:28,688 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcCVV2M/build/release/bin/postgresql.jar:/tmp/dist-test-taskcCVV2M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.225.148:48769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 06:29:28,947 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 06:29:28,947 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcCVV2M/build/release/bin/postgresql.jar:/tmp/dist-test-taskcCVV2M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.225.148:48769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 06:29:29,241 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 06:29:29,241 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcCVV2M/build/release/bin/postgresql.jar:/tmp/dist-test-taskcCVV2M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.225.148:48769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 06:29:29,511 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcCVV2M/build/release/bin/postgresql.jar:/tmp/dist-test-taskcCVV2M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.225.148:48769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 06:29:29,806 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcCVV2M/build/release/bin/postgresql.jar:/tmp/dist-test-taskcCVV2M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.225.148:48769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-d59m.c.gcp-upstream.internal';"
2026-09-28 06:29:30,102 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 06:29:30,102 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcCVV2M/build/release/bin/postgresql.jar:/tmp/dist-test-taskcCVV2M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.225.148:48769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 06:29:30,373 [I] DB_PATCHES have already been applied
I20260928 06:29:30.378739 23430 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/ranger-admin:/tmp/dist-test-taskcCVV2M/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskcCVV2M/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskcCVV2M/build/release/bin/postgresql.jar:/tmp/dist-test-taskcCVV2M/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 06:29:30.378822 23430 mini_ranger.cc:193] Using host: 127.22.225.148
I20260928 06:29:30.382131 23430 mini_ranger.cc:227] Ranger admin URL: http://127.22.225.148:56545
Sep 28, 2026 6:29:30 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 6:29:30 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 6:29:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 6:29:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 6:29:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 6:29:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 6:29:31 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 06:29:32.753688 23430 mini_ranger.cc:160] Time spent starting Ranger: real 10.964s user 0.003s sys 0.009s
Sep 28, 2026 6:29:58 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 6:29:58 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 28, 2026 6:29:58 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 6:29:58 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 6:29:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 6:29:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 6:29:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 6:29:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 6:29:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 6:29:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 6:29:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 6:29:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 6:29:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 6:29:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 6:29:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 6:29:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 6:29:58 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 6:29:59 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 06:30:00.358258 23430 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.22.225.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.22.225.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.22.225.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/krb5kdc/rangerkms_127.22.225.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.22.225.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/krb5kdc/rangerkms_127.22.225.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.22.225.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.22.225.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.225.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/krb5kdc/HTTP_127.22.225.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.22.225.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/krb5kdc/HTTP_127.22.225.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 06:30:00 dist-test-slave-d59m krb5kdc[1705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577000, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 06:30:00.448634 23430 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 06:30:00.460898 23430 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 06:30:01.126824 23430 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 06:30:01.126915 23430 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/ranger-kms
I20260928 06:30:01.126935 23430 mini_ranger_kms.cc:227] Using postgres at 127.22.225.148:48769
2026-09-28 06:30:01,859 [I] DB FLAVOR :POSTGRES
2026-09-28 06:30:01,859 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 06:30:01,859 [I] Checking connection
2026-09-28 06:30:01,859 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcCVV2M/build/release/bin/postgresql.jar:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.225.148:48769/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 06:30:02,123 [I] connection success
2026-09-28 06:30:02,123 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 06:30:02,123 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 06:30:02,123 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcCVV2M/build/release/bin/postgresql.jar:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.225.148:48769/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 06:30:02,393 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 06:30:02,393 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 06:30:02,393 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 06:30:02,393 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcCVV2M/build/release/bin/postgresql.jar:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.225.148:48769/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 06:30:02,691 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 06:30:03.230592 23430 mini_ranger_kms.cc:310] Created kms service
I20260928 06:30:03.444252 23430 mini_ranger_kms.cc:326] Created kudu user
I20260928 06:30:03.510501 23430 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 06:30:03 dist-test-slave-d59m krb5kdc[1705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577000, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.22.225.148@KRBTEST.COM
I20260928 06:30:03.994963 23430 mini_ranger_kms.cc:384] Added ranger policy
I20260928 06:30:03.995082 23430 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/ranger-kms:/tmp/dist-test-taskcCVV2M/build/release/bin/postgresql.jar:/tmp/dist-test-taskcCVV2M/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskcCVV2M/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 06:30:03.995103 23430 mini_ranger_kms.cc:240] Using host: 127.22.225.148
I20260928 06:30:04.004243 23430 mini_ranger_kms.cc:276] Ranger KMS PID: 2100
I20260928 06:30:04.004410 23430 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.22.225.148:53217
Sep 28, 2026 6:30:05 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 6:30:05 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 6:30:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 6:30:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 6:30:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 6:30:06 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 06:30:06.421903 23430 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.973s user 0.005s sys 0.265s
I20260928 06:30:06.422005 23430 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 06:30:06.422077 23430 mini_ranger_kms.cc:401] 127.22.225.148:53217/kms/v1/keys
Sep 28 06:30:07 dist-test-slave-d59m krb5kdc[1705](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790577007, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.225.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 06:30:07 dist-test-slave-d59m krb5kdc[1705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577007, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.225.148@KRBTEST.COM for HTTP/127.22.225.148@KRBTEST.COM
Sep 28 06:30:08 dist-test-slave-d59m krb5kdc[1705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577007, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.225.148@KRBTEST.COM for HTTP/127.22.225.148@KRBTEST.COM
Sep 28 06:30:10 dist-test-slave-d59m krb5kdc[1705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.22.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab.
Entry for principal kudu/127.22.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab.
I20260928 06:30:10.831823 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:45141
--webserver_interface=127.22.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab
--principal=kudu/127.22.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:33609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.225.190:45141
--ranger_config_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.225.148:53217/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:30:10.941208 2150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:30:10.941491 2150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:30:10.941525 2150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:30:10.943534 2150 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:30:10.943598 2150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:30:10.943617 2150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:30:10.943636 2150 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:30:10.943665 2150 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:30:10.946009 2150 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:33609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/master-0/wal
--ranger_kms_url=127.22.225.148:53217/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.22.225.190:45141
--ranger_config_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab
--principal=kudu/127.22.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:45141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:30:10.946261 2150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:30:10.946563 2150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:30:10.952927 2155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:30:10.952939 2156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:30:10.953342 2158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:30:10.953456 2150 server_base.cc:1034] running on GCE node
I20260928 06:30:10.953918 2150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:30:10.954300 2150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:30:10.955498 2150 hybrid_clock.cc:648] HybridClock initialized: now 1790577010955470 us; error 46 us; skew 500 ppm
Sep 28 06:30:10 dist-test-slave-d59m krb5kdc[1705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577010, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:30:10.958389 2150 init.cc:377] Logged in from keytab as kudu/127.22.225.190@KRBTEST.COM (short username kudu)
I20260928 06:30:10.958956 2150 webserver.cc:460] Webserver started at http://127.22.225.190:34511/ using document root <none> and password file <none>
I20260928 06:30:10.959219 2150 fs_manager.cc:359] Metadata directory not provided
I20260928 06:30:10.959275 2150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:30:10.959471 2150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 06:30:10 dist-test-slave-d59m krb5kdc[1705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577010, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.190@KRBTEST.COM for HTTP/127.22.225.148@KRBTEST.COM
I20260928 06:30:11.021926 2150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/master-0/data/instance:
uuid: "ae4a5f0e447c475584d6f20c69a53d47"
format_stamp: "Formatted at 2026-09-28 06:30:11 on dist-test-slave-d59m"
server_key: "11d98c7fc5632fc56a3f6c812f5cbf8e"
server_key_iv: "205aabc14609d98290241df53614cbc0"
server_key_version: "kuduclusterkey@0"
I20260928 06:30:11.022428 2150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/master-0/wal/instance:
uuid: "ae4a5f0e447c475584d6f20c69a53d47"
format_stamp: "Formatted at 2026-09-28 06:30:11 on dist-test-slave-d59m"
server_key: "11d98c7fc5632fc56a3f6c812f5cbf8e"
server_key_iv: "205aabc14609d98290241df53614cbc0"
server_key_version: "kuduclusterkey@0"
I20260928 06:30:11.024047 2150 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:30:11.040274 2165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:30:11.040469 2150 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260928 06:30:11.040563 2150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/master-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/master-0/wal
uuid: "ae4a5f0e447c475584d6f20c69a53d47"
format_stamp: "Formatted at 2026-09-28 06:30:11 on dist-test-slave-d59m"
server_key: "11d98c7fc5632fc56a3f6c812f5cbf8e"
server_key_iv: "205aabc14609d98290241df53614cbc0"
server_key_version: "kuduclusterkey@0"
I20260928 06:30:11.040731 2150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:30:11.054761 2150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:30:11.056864 2150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:30:11.057037 2150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:30:11.062292 2150 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.190:45141
I20260928 06:30:11.062356 2217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.190:45141 every 8 connection(s)
I20260928 06:30:11.062759 2150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
I20260928 06:30:11.063602 2218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:30:11.066876 2218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae4a5f0e447c475584d6f20c69a53d47: Bootstrap starting.
I20260928 06:30:11.067560 2218 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae4a5f0e447c475584d6f20c69a53d47: Neither blocks nor log segments found. Creating new log.
I20260928 06:30:11.067905 2218 log.cc:824] T 00000000000000000000000000000000 P ae4a5f0e447c475584d6f20c69a53d47: Log is configured to *not* fsync() on all Append() calls
I20260928 06:30:11.068257 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 2150
I20260928 06:30:11.068367 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/master-0/wal/instance
I20260928 06:30:11.068569 23430 external_mini_cluster.cc:1428] Setting key 3bf3a655ef4905ef401546ab057695a4
I20260928 06:30:11.068828 2218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae4a5f0e447c475584d6f20c69a53d47: No bootstrap required, opened a new log
I20260928 06:30:11.071993 2218 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae4a5f0e447c475584d6f20c69a53d47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae4a5f0e447c475584d6f20c69a53d47" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 45141 } }
I20260928 06:30:11.072216 2218 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae4a5f0e447c475584d6f20c69a53d47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:30:11.072304 2218 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae4a5f0e447c475584d6f20c69a53d47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae4a5f0e447c475584d6f20c69a53d47, State: Initialized, Role: FOLLOWER
I20260928 06:30:11.072474 2218 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae4a5f0e447c475584d6f20c69a53d47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae4a5f0e447c475584d6f20c69a53d47" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 45141 } }
I20260928 06:30:11.072621 2218 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae4a5f0e447c475584d6f20c69a53d47 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:30:11.072669 2218 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae4a5f0e447c475584d6f20c69a53d47 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:30:11.072722 2218 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae4a5f0e447c475584d6f20c69a53d47 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:30:11.073335 2218 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae4a5f0e447c475584d6f20c69a53d47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae4a5f0e447c475584d6f20c69a53d47" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 45141 } }
I20260928 06:30:11.073446 2218 leader_election.cc:302] T 00000000000000000000000000000000 P ae4a5f0e447c475584d6f20c69a53d47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae4a5f0e447c475584d6f20c69a53d47; no voters:
I20260928 06:30:11.073673 2218 leader_election.cc:288] T 00000000000000000000000000000000 P ae4a5f0e447c475584d6f20c69a53d47 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:30:11.073701 2223 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae4a5f0e447c475584d6f20c69a53d47 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:30:11.073810 2223 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae4a5f0e447c475584d6f20c69a53d47 [term 1 LEADER]: Becoming Leader. State: Replica: ae4a5f0e447c475584d6f20c69a53d47, State: Running, Role: LEADER
I20260928 06:30:11.073890 2223 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae4a5f0e447c475584d6f20c69a53d47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae4a5f0e447c475584d6f20c69a53d47" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 45141 } }
I20260928 06:30:11.073953 2218 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae4a5f0e447c475584d6f20c69a53d47 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:30:11.074342 2225 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae4a5f0e447c475584d6f20c69a53d47 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae4a5f0e447c475584d6f20c69a53d47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae4a5f0e447c475584d6f20c69a53d47" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 45141 } } }
I20260928 06:30:11.074503 2225 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae4a5f0e447c475584d6f20c69a53d47 [sys.catalog]: This master's current role is: LEADER
I20260928 06:30:11.074718 2224 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae4a5f0e447c475584d6f20c69a53d47 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae4a5f0e447c475584d6f20c69a53d47. Latest consensus state: current_term: 1 leader_uuid: "ae4a5f0e447c475584d6f20c69a53d47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae4a5f0e447c475584d6f20c69a53d47" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 45141 } } }
I20260928 06:30:11.074792 2229 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:30:11.075366 2229 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:30:11.075640 2218 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 06:30:11.077589 2224 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae4a5f0e447c475584d6f20c69a53d47 [sys.catalog]: This master's current role is: LEADER
Sep 28 06:30:11 dist-test-slave-d59m krb5kdc[1705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:30:11.084384 2229 catalog_manager.cc:1347] Generated new cluster ID: 667d1bae9e464fa0983033999da859c5
I20260928 06:30:11.084450 2229 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:30:11.087160 2222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:30:11.069216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53470 (local address 127.22.225.190:45141)
0928 06:30:11.069379 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 06:30:11.069390 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:30:11.069424 (+ 34us) server_negotiation.cc:407] Connection header received
0928 06:30:11.071972 (+ 2548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:30:11.071980 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:30:11.072035 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:30:11.072204 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:30:11.073101 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:30:11.074414 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:30:11.075515 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:30:11.078450 (+ 2935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:30:11.081837 (+ 3387us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:30:11.082014 (+ 177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:30:11.082017 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:30:11.082017 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:30:11.082027 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:30:11.084942 (+ 2915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:30:11.085394 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:30:11.085396 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:30:11.085397 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:30:11.085509 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:30:11.085735 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:30:11.085736 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:30:11.085736 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:30:11.086076 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:30:11.086626 (+ 550us) server_negotiation.cc:1035] Waiting for connection context
0928 06:30:11.086710 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 06:30:11.086871 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":78,"threads_started":1}
I20260928 06:30:11.090929 2229 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:30:11.091629 2229 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:30:11.100873 2229 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae4a5f0e447c475584d6f20c69a53d47: Generated new TSK 0
I20260928 06:30:11.101140 2229 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.225.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab.
Entry for principal kudu/127.22.225.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab.
I20260928 06:30:11.973395 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.225.148:53217/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.129:0
--local_ip_for_outbound_sockets=127.22.225.129
--webserver_interface=127.22.225.129
--webserver_port=0
--tserver_master_addrs=127.22.225.190:45141
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab
--principal=kudu/127.22.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:33609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:30:12.092823 2264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:30:12.093123 2264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:30:12.093220 2264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:30:12.095259 2264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:30:12.095316 2264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:30:12.095355 2264 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.129
I20260928 06:30:12.097656 2264 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:33609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.22.225.148:53217/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab
--principal=kudu/127.22.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.225.129
--webserver_port=0
--tserver_master_addrs=127.22.225.190:45141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2264
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.129
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:30:12.098094 2264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:30:12.098452 2264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:30:12.105710 2272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:30:12.106127 2269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:30:12.107105 2270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:30:12.107744 2264 server_base.cc:1034] running on GCE node
I20260928 06:30:12.107968 2264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:30:12.108260 2264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:30:12.109447 2264 hybrid_clock.cc:648] HybridClock initialized: now 1790577012109426 us; error 32 us; skew 500 ppm
Sep 28 06:30:12 dist-test-slave-d59m krb5kdc[1705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577012, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:30:12.113339 2264 init.cc:377] Logged in from keytab as kudu/127.22.225.129@KRBTEST.COM (short username kudu)
I20260928 06:30:12.113973 2264 webserver.cc:460] Webserver started at http://127.22.225.129:46501/ using document root <none> and password file <none>
I20260928 06:30:12.114306 2264 fs_manager.cc:359] Metadata directory not provided
I20260928 06:30:12.114498 2264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:30:12.114709 2264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 06:30:12 dist-test-slave-d59m krb5kdc[1705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577012, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.129@KRBTEST.COM for HTTP/127.22.225.148@KRBTEST.COM
I20260928 06:30:12.131671 2264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/ts-0/data/instance:
uuid: "775328fa003c413290f840c989024f57"
format_stamp: "Formatted at 2026-09-28 06:30:12 on dist-test-slave-d59m"
server_key: "4ca2fc67443f005561be280af25e465c"
server_key_iv: "1d9ef0d8cc9b5a76b147e936c0cb1b07"
server_key_version: "kuduclusterkey@0"
I20260928 06:30:12.132148 2264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/ts-0/wal/instance:
uuid: "775328fa003c413290f840c989024f57"
format_stamp: "Formatted at 2026-09-28 06:30:12 on dist-test-slave-d59m"
server_key: "4ca2fc67443f005561be280af25e465c"
server_key_iv: "1d9ef0d8cc9b5a76b147e936c0cb1b07"
server_key_version: "kuduclusterkey@0"
I20260928 06:30:12.134707 2264 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260928 06:30:12.159459 2280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:30:12.163432 2264 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260928 06:30:12.163565 2264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/ts-0/wal
uuid: "775328fa003c413290f840c989024f57"
format_stamp: "Formatted at 2026-09-28 06:30:12 on dist-test-slave-d59m"
server_key: "4ca2fc67443f005561be280af25e465c"
server_key_iv: "1d9ef0d8cc9b5a76b147e936c0cb1b07"
server_key_version: "kuduclusterkey@0"
I20260928 06:30:12.163668 2264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:30:12.193357 2264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:30:12.197181 2264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:30:12.197418 2264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:30:12.197739 2264 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:30:12.198274 2264 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:30:12.198374 2264 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:30:12.198472 2264 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:30:12.198545 2264 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:30:12.205034 2264 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.129:41669
I20260928 06:30:12.205534 2264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
I20260928 06:30:12.206615 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 2264
I20260928 06:30:12.206734 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/ts-0/wal/instance
I20260928 06:30:12.206938 23430 external_mini_cluster.cc:1428] Setting key 6688d64d6e152a7f4b940220d8746c76
I20260928 06:30:12.214468 2394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.129:41669 every 8 connection(s)
WARNING: no policy specified for kudu/127.22.225.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.130@KRBTEST.COM" created.
Sep 28 06:30:12 dist-test-slave-d59m krb5kdc[1705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577012, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.129@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:30:12.237979 2398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:30:12.215649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:45141 (local address 127.22.225.129:35525)
0928 06:30:12.218478 (+ 2829us) negotiation.cc:107] Waiting for socket to connect
0928 06:30:12.218532 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 06:30:12.219326 (+ 794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:30:12.219953 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:30:12.219958 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:30:12.220248 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:30:12.220532 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:30:12.220539 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:30:12.223584 (+ 3045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:30:12.223585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:30:12.225003 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:30:12.225006 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:30:12.225579 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:30:12.225580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:30:12.225758 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:30:12.227439 (+ 1681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:30:12.227450 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:30:12.229174 (+ 1724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:30:12.236148 (+ 6974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:30:12.236158 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:30:12.236161 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:30:12.236479 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:30:12.236798 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:30:12.236799 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:30:12.236799 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:30:12.236953 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:30:12.237258 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:30:12.237265 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:30:12.237441 (+ 176us) client_negotiation.cc:769] Sending connection context
0928 06:30:12.237548 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 06:30:12.237727 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2769,"thread_start_us":138,"threads_started":1}
I20260928 06:30:12.238588 2399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:30:12.216032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:35525 (local address 127.22.225.190:45141)
0928 06:30:12.219687 (+ 3655us) server_negotiation.cc:206] Beginning negotiation
0928 06:30:12.219690 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:30:12.219699 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:30:12.219727 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:30:12.219728 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:30:12.219740 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:30:12.219807 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:30:12.220832 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:30:12.223464 (+ 2632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:30:12.225107 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:30:12.225433 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:30:12.225501 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:30:12.229295 (+ 3794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:30:12.229297 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:30:12.229297 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:30:12.229306 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:30:12.236033 (+ 6727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:30:12.236636 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:30:12.236638 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:30:12.236638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:30:12.236719 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:30:12.237024 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:30:12.237024 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:30:12.237025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:30:12.237169 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:30:12.237231 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 06:30:12.238385 (+ 1154us) server_negotiation.cc:299] Negotiation successful
0928 06:30:12.238481 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3619,"thread_start_us":51,"threads_started":1}
I20260928 06:30:12.239341 2395 heartbeater.cc:347] Connected to a master server at 127.22.225.190:45141
I20260928 06:30:12.239437 2395 heartbeater.cc:464] Registering TS with master...
I20260928 06:30:12.239640 2395 heartbeater.cc:511] Master 127.22.225.190:45141 requested a full tablet report, sending...
I20260928 06:30:12.240255 2182 ts_manager.cc:194] Registered new tserver with Master: 775328fa003c413290f840c989024f57 (127.22.225.129:41669)
I20260928 06:30:12.241155 2182 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.225.129@KRBTEST.COM'} at 127.22.225.129:35525
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab.
Entry for principal kudu/127.22.225.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab.
I20260928 06:30:12.266151 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.225.148:53217/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.130:0
--local_ip_for_outbound_sockets=127.22.225.130
--webserver_interface=127.22.225.130
--webserver_port=0
--tserver_master_addrs=127.22.225.190:45141
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab
--principal=kudu/127.22.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:33609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:30:12.387931 2401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:30:12.388302 2401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:30:12.388376 2401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:30:12.390048 2401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:30:12.390157 2401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:30:12.390249 2401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.130
I20260928 06:30:12.392414 2401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:33609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.22.225.148:53217/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab
--principal=kudu/127.22.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.225.130
--webserver_port=0
--tserver_master_addrs=127.22.225.190:45141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2401
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.130
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:30:12.392882 2401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:30:12.393208 2401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:30:12.405627 2408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:30:12.406078 2411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:30:12.408437 2401 server_base.cc:1034] running on GCE node
W20260928 06:30:12.409610 2409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:30:12.409938 2401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:30:12.410267 2401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:30:12.415409 2401 hybrid_clock.cc:648] HybridClock initialized: now 1790577012415380 us; error 46 us; skew 500 ppm
Sep 28 06:30:12 dist-test-slave-d59m krb5kdc[1705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577012, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:30:12.418797 2401 init.cc:377] Logged in from keytab as kudu/127.22.225.130@KRBTEST.COM (short username kudu)
I20260928 06:30:12.419359 2401 webserver.cc:460] Webserver started at http://127.22.225.130:42923/ using document root <none> and password file <none>
I20260928 06:30:12.419613 2401 fs_manager.cc:359] Metadata directory not provided
I20260928 06:30:12.419665 2401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:30:12.419880 2401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 06:30:12 dist-test-slave-d59m krb5kdc[1705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577012, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.130@KRBTEST.COM for HTTP/127.22.225.148@KRBTEST.COM
Sep 28 06:30:12 dist-test-slave-d59m krb5kdc[1705](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790577012, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:30:12.437875 2401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/ts-1/data/instance:
uuid: "6f041c2e4531499f8d180c6d49e8e4ae"
format_stamp: "Formatted at 2026-09-28 06:30:12 on dist-test-slave-d59m"
server_key: "08f166faae5271e0d579dec34f2b5f4e"
server_key_iv: "30b3d2f834f38e26bb12d6cfcb616bcf"
server_key_version: "kuduclusterkey@0"
I20260928 06:30:12.438429 2401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/ts-1/wal/instance:
uuid: "6f041c2e4531499f8d180c6d49e8e4ae"
format_stamp: "Formatted at 2026-09-28 06:30:12 on dist-test-slave-d59m"
server_key: "08f166faae5271e0d579dec34f2b5f4e"
server_key_iv: "30b3d2f834f38e26bb12d6cfcb616bcf"
server_key_version: "kuduclusterkey@0"
I20260928 06:30:12.440665 2401 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.001s
I20260928 06:30:12.469300 2418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:30:12.469484 2401 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:30:12.469579 2401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/ts-1/wal
uuid: "6f041c2e4531499f8d180c6d49e8e4ae"
format_stamp: "Formatted at 2026-09-28 06:30:12 on dist-test-slave-d59m"
server_key: "08f166faae5271e0d579dec34f2b5f4e"
server_key_iv: "30b3d2f834f38e26bb12d6cfcb616bcf"
server_key_version: "kuduclusterkey@0"
I20260928 06:30:12.469700 2401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:30:12.497825 2401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:30:12.500134 2401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:30:12.500320 2401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:30:12.500656 2401 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:30:12.501129 2401 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:30:12.501187 2401 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:30:12.501228 2401 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:30:12.501255 2401 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:30:12.510232 2401 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.130:38165
I20260928 06:30:12.510705 2401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
I20260928 06:30:12.511232 2531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.130:38165 every 8 connection(s)
I20260928 06:30:12.517323 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 2401
I20260928 06:30:12.517432 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/ts-1/wal/instance
I20260928 06:30:12.517623 23430 external_mini_cluster.cc:1428] Setting key 22db4cd084785bcaff53f4e965017564
Sep 28 06:30:12 dist-test-slave-d59m krb5kdc[1705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577012, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.130@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:30:12.524086 2399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:30:12.512197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:49081 (local address 127.22.225.190:45141)
0928 06:30:12.512282 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0928 06:30:12.512285 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:30:12.512297 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:30:12.512937 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:30:12.512938 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:30:12.512955 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:30:12.513018 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:30:12.513632 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:30:12.514140 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:30:12.515248 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:30:12.515471 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:30:12.515564 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:30:12.520518 (+ 4954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:30:12.520520 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:30:12.520520 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:30:12.520529 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:30:12.522343 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:30:12.522873 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:30:12.522875 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:30:12.522875 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:30:12.522939 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:30:12.523274 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:30:12.523275 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:30:12.523275 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:30:12.523425 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:30:12.523474 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 06:30:12.523884 (+ 410us) server_negotiation.cc:299] Negotiation successful
0928 06:30:12.523982 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 06:30:12.524657 2534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:30:12.511862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:45141 (local address 127.22.225.130:49081)
0928 06:30:12.512063 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 06:30:12.512108 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 06:30:12.512788 (+ 680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:30:12.513093 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:30:12.513097 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:30:12.513296 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:30:12.513519 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:30:12.513523 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:30:12.514261 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:30:12.514262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:30:12.515129 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:30:12.515131 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:30:12.515584 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:30:12.515585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:30:12.515773 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:30:12.516427 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:30:12.516444 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:30:12.520389 (+ 3945us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:30:12.522444 (+ 2055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:30:12.522452 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:30:12.522455 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:30:12.522773 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:30:12.523019 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:30:12.523020 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:30:12.523020 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:30:12.523182 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:30:12.523526 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:30:12.523533 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:30:12.523710 (+ 177us) client_negotiation.cc:769] Sending connection context
0928 06:30:12.524202 (+ 492us) client_negotiation.cc:240] Negotiation successful
0928 06:30:12.524387 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":111,"threads_started":1}
I20260928 06:30:12.525131 2532 heartbeater.cc:347] Connected to a master server at 127.22.225.190:45141
I20260928 06:30:12.525249 2532 heartbeater.cc:464] Registering TS with master...
I20260928 06:30:12.525476 2532 heartbeater.cc:511] Master 127.22.225.190:45141 requested a full tablet report, sending...
I20260928 06:30:12.525878 2182 ts_manager.cc:194] Registered new tserver with Master: 6f041c2e4531499f8d180c6d49e8e4ae (127.22.225.130:38165)
I20260928 06:30:12.526368 2182 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.225.130@KRBTEST.COM'} at 127.22.225.130:49081
WARNING: no policy specified for kudu/127.22.225.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab.
Entry for principal kudu/127.22.225.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab.
I20260928 06:30:12.568543 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.225.148:53217/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.131:0
--local_ip_for_outbound_sockets=127.22.225.131
--webserver_interface=127.22.225.131
--webserver_port=0
--tserver_master_addrs=127.22.225.190:45141
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab
--principal=kudu/127.22.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:33609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:30:12.691179 2538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:30:12.691520 2538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:30:12.691612 2538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:30:12.693228 2538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:30:12.693329 2538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:30:12.693411 2538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.131
I20260928 06:30:12.695499 2538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:33609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.22.225.148:53217/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab
--principal=kudu/127.22.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.225.131
--webserver_port=0
--tserver_master_addrs=127.22.225.190:45141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2538
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.131
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:30:12.695958 2538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:30:12.696273 2538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:30:12.702718 2544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:30:12.706333 2538 server_base.cc:1034] running on GCE node
W20260928 06:30:12.706614 2543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:30:12.709321 2546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:30:12.709790 2538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:30:12.710130 2538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:30:12.718896 2538 hybrid_clock.cc:648] HybridClock initialized: now 1790577012718863 us; error 40 us; skew 500 ppm
Sep 28 06:30:12 dist-test-slave-d59m krb5kdc[1705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577012, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:30:12.722810 2538 init.cc:377] Logged in from keytab as kudu/127.22.225.131@KRBTEST.COM (short username kudu)
I20260928 06:30:12.723428 2538 webserver.cc:460] Webserver started at http://127.22.225.131:41775/ using document root <none> and password file <none>
I20260928 06:30:12.723738 2538 fs_manager.cc:359] Metadata directory not provided
I20260928 06:30:12.723850 2538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:30:12.724025 2538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 06:30:12 dist-test-slave-d59m krb5kdc[1705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577012, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.131@KRBTEST.COM for HTTP/127.22.225.148@KRBTEST.COM
I20260928 06:30:12.737870 2538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/ts-2/data/instance:
uuid: "1011ee6c88d94894b5f2625d83c4f028"
format_stamp: "Formatted at 2026-09-28 06:30:12 on dist-test-slave-d59m"
server_key: "b6f9df4a146c1fe4fbd27adfe0bb0871"
server_key_iv: "3a508cc9a5fc37643d484f06162e7f27"
server_key_version: "kuduclusterkey@0"
I20260928 06:30:12.738345 2538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/ts-2/wal/instance:
uuid: "1011ee6c88d94894b5f2625d83c4f028"
format_stamp: "Formatted at 2026-09-28 06:30:12 on dist-test-slave-d59m"
server_key: "b6f9df4a146c1fe4fbd27adfe0bb0871"
server_key_iv: "3a508cc9a5fc37643d484f06162e7f27"
server_key_version: "kuduclusterkey@0"
I20260928 06:30:12.741529 2538 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260928 06:30:12.767931 2553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:30:12.768177 2538 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.001s sys 0.001s
I20260928 06:30:12.768282 2538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/ts-2/wal
uuid: "1011ee6c88d94894b5f2625d83c4f028"
format_stamp: "Formatted at 2026-09-28 06:30:12 on dist-test-slave-d59m"
server_key: "b6f9df4a146c1fe4fbd27adfe0bb0871"
server_key_iv: "3a508cc9a5fc37643d484f06162e7f27"
server_key_version: "kuduclusterkey@0"
I20260928 06:30:12.768388 2538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:30:12.788044 2538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:30:12.790566 2538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:30:12.790735 2538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:30:12.790943 2538 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:30:12.791320 2538 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:30:12.791366 2538 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:30:12.791407 2538 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:30:12.791432 2538 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:30:12.804852 2538 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.131:43167
I20260928 06:30:12.805373 2538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
I20260928 06:30:12.806740 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 2538
I20260928 06:30:12.806847 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/ts-2/wal/instance
I20260928 06:30:12.807044 23430 external_mini_cluster.cc:1428] Setting key 9cd3f5603e4635ced1f850f5ca91225b
I20260928 06:30:12.817044 2666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.131:43167 every 8 connection(s)
Sep 28 06:30:12 dist-test-slave-d59m krb5kdc[1705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577012, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.131@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:30:12.847720 2669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:30:12.817672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:45141 (local address 127.22.225.131:54931)
0928 06:30:12.818015 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0928 06:30:12.818063 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 06:30:12.819031 (+ 968us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:30:12.819396 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:30:12.819400 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:30:12.819648 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:30:12.820003 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:30:12.820010 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:30:12.820970 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:30:12.820971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:30:12.831142 (+ 10171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:30:12.831146 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:30:12.831698 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:30:12.831699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:30:12.831844 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:30:12.841518 (+ 9674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:30:12.841532 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:30:12.843567 (+ 2035us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:30:12.845714 (+ 2147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:30:12.845726 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:30:12.845729 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:30:12.846078 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:30:12.846343 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:30:12.846344 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:30:12.846344 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:30:12.846507 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:30:12.846857 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:30:12.846864 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:30:12.847045 (+ 181us) client_negotiation.cc:769] Sending connection context
0928 06:30:12.847174 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 06:30:12.847352 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":120,"threads_started":1}
I20260928 06:30:12.848253 2399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:30:12.817738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:54931 (local address 127.22.225.190:45141)
0928 06:30:12.817805 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 06:30:12.817808 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:30:12.818156 (+ 348us) server_negotiation.cc:407] Connection header received
0928 06:30:12.819166 (+ 1010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:30:12.819167 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:30:12.819180 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:30:12.819211 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:30:12.820116 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:30:12.820856 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:30:12.831282 (+ 10426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:30:12.831552 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:30:12.831596 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:30:12.843721 (+ 12125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:30:12.843723 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:30:12.843723 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:30:12.843732 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:30:12.845577 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:30:12.846196 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:30:12.846197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:30:12.846197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:30:12.846263 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:30:12.846588 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:30:12.846589 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:30:12.846589 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:30:12.846761 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:30:12.846827 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 06:30:12.848078 (+ 1251us) server_negotiation.cc:299] Negotiation successful
0928 06:30:12.848161 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 06:30:12.848985 2667 heartbeater.cc:347] Connected to a master server at 127.22.225.190:45141
I20260928 06:30:12.849107 2667 heartbeater.cc:464] Registering TS with master...
I20260928 06:30:12.849319 2667 heartbeater.cc:511] Master 127.22.225.190:45141 requested a full tablet report, sending...
I20260928 06:30:12.849686 2182 ts_manager.cc:194] Registered new tserver with Master: 1011ee6c88d94894b5f2625d83c4f028 (127.22.225.131:43167)
I20260928 06:30:12.850212 2182 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.225.131@KRBTEST.COM'} at 127.22.225.131:54931
I20260928 06:30:12.854323 23430 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:30:12.866101 2399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:30:12.856171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53486 (local address 127.22.225.190:45141)
0928 06:30:12.856212 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 06:30:12.856214 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:30:12.856823 (+ 609us) server_negotiation.cc:407] Connection header received
0928 06:30:12.856925 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:30:12.856926 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:30:12.856938 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:30:12.856967 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:30:12.857923 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:30:12.858708 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:30:12.859870 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:30:12.861033 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:30:12.861085 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:30:12.862902 (+ 1817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:30:12.862903 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:30:12.862904 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:30:12.862911 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:30:12.864703 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:30:12.865139 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:30:12.865140 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:30:12.865140 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:30:12.865198 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:30:12.865424 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:30:12.865425 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:30:12.865426 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:30:12.865574 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:30:12.865679 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 06:30:12.865920 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 06:30:12.866008 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:30:12.868681 2182 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53486:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:30:12.871291 2182 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:30:12.892159 2680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:30:12.884364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.130:38165 (local address 127.0.0.1:51386)
0928 06:30:12.886351 (+ 1987us) negotiation.cc:107] Waiting for socket to connect
0928 06:30:12.886361 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:30:12.886408 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:30:12.887023 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:30:12.887027 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:30:12.887029 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:30:12.887108 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:30:12.887110 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:30:12.888950 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:30:12.888951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:30:12.890636 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:30:12.890640 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:30:12.891834 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:30:12.891835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:30:12.891917 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:30:12.891931 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 06:30:12.892003 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 06:30:12.892072 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1964,"thread_start_us":160,"threads_started":1}
I20260928 06:30:12.893980 2683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:30:12.884984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51386 (local address 127.22.225.130:38165)
0928 06:30:12.885952 (+ 968us) server_negotiation.cc:206] Beginning negotiation
0928 06:30:12.885958 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:30:12.886465 (+ 507us) server_negotiation.cc:407] Connection header received
0928 06:30:12.886525 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:30:12.886528 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:30:12.886545 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:30:12.886664 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:30:12.887197 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:30:12.888849 (+ 1652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:30:12.890777 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:30:12.891583 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:30:12.891696 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:30:12.891758 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 06:30:12.893852 (+ 2094us) server_negotiation.cc:299] Negotiation successful
0928 06:30:12.893898 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":910,"thread_start_us":140,"threads_started":1}
I20260928 06:30:12.894974 2466 tablet_service.cc:1467] Processing CreateTablet for tablet 72997cc4ce734d5d80f29969c5f9ed02 (DEFAULT_TABLE table=test-table [id=712316e49d4d42cf930b8bd6f1ef1df0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:30:12.895381 2466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72997cc4ce734d5d80f29969c5f9ed02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:30:12.897266 2684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:30:12.885349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.129:41669 (local address 127.0.0.1:54242)
0928 06:30:12.886871 (+ 1522us) negotiation.cc:107] Waiting for socket to connect
0928 06:30:12.886882 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:30:12.886934 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:30:12.892921 (+ 5987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:30:12.892922 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:30:12.892924 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:30:12.892965 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:30:12.892966 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:30:12.894786 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:30:12.894787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:30:12.895876 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:30:12.895879 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:30:12.896984 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:30:12.896985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:30:12.897047 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:30:12.897051 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:30:12.897118 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 06:30:12.897161 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1481,"thread_start_us":143,"threads_started":1}
I20260928 06:30:12.899211 2685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:30:12.885471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54242 (local address 127.22.225.129:41669)
0928 06:30:12.892680 (+ 7209us) server_negotiation.cc:206] Beginning negotiation
0928 06:30:12.892686 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:30:12.892698 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:30:12.892736 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:30:12.892737 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:30:12.892754 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:30:12.892846 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:30:12.893033 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:30:12.894169 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:30:12.895993 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:30:12.896807 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:30:12.896884 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:30:12.896947 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 06:30:12.899079 (+ 2132us) server_negotiation.cc:299] Negotiation successful
0928 06:30:12.899144 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7163,"thread_start_us":52,"threads_started":1}
I20260928 06:30:12.899901 2329 tablet_service.cc:1467] Processing CreateTablet for tablet 72997cc4ce734d5d80f29969c5f9ed02 (DEFAULT_TABLE table=test-table [id=712316e49d4d42cf930b8bd6f1ef1df0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:30:12.900272 2686 tablet_bootstrap.cc:492] T 72997cc4ce734d5d80f29969c5f9ed02 P 6f041c2e4531499f8d180c6d49e8e4ae: Bootstrap starting.
I20260928 06:30:12.900301 2329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72997cc4ce734d5d80f29969c5f9ed02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:30:12.901894 2686 tablet_bootstrap.cc:654] T 72997cc4ce734d5d80f29969c5f9ed02 P 6f041c2e4531499f8d180c6d49e8e4ae: Neither blocks nor log segments found. Creating new log.
I20260928 06:30:12.902326 2686 log.cc:824] T 72997cc4ce734d5d80f29969c5f9ed02 P 6f041c2e4531499f8d180c6d49e8e4ae: Log is configured to *not* fsync() on all Append() calls
I20260928 06:30:12.903496 2686 tablet_bootstrap.cc:492] T 72997cc4ce734d5d80f29969c5f9ed02 P 6f041c2e4531499f8d180c6d49e8e4ae: No bootstrap required, opened a new log
I20260928 06:30:12.903563 2686 ts_tablet_manager.cc:1397] T 72997cc4ce734d5d80f29969c5f9ed02 P 6f041c2e4531499f8d180c6d49e8e4ae: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:30:12.906396 2686 raft_consensus.cc:348] T 72997cc4ce734d5d80f29969c5f9ed02 P 6f041c2e4531499f8d180c6d49e8e4ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "775328fa003c413290f840c989024f57" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 41669 } } peers { permanent_uuid: "1011ee6c88d94894b5f2625d83c4f028" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 43167 } } peers { permanent_uuid: "6f041c2e4531499f8d180c6d49e8e4ae" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 38165 } }
I20260928 06:30:12.906553 2686 raft_consensus.cc:374] T 72997cc4ce734d5d80f29969c5f9ed02 P 6f041c2e4531499f8d180c6d49e8e4ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:30:12.906584 2686 raft_consensus.cc:729] T 72997cc4ce734d5d80f29969c5f9ed02 P 6f041c2e4531499f8d180c6d49e8e4ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f041c2e4531499f8d180c6d49e8e4ae, State: Initialized, Role: FOLLOWER
I20260928 06:30:12.907466 2686 consensus_queue.cc:260] T 72997cc4ce734d5d80f29969c5f9ed02 P 6f041c2e4531499f8d180c6d49e8e4ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "775328fa003c413290f840c989024f57" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 41669 } } peers { permanent_uuid: "1011ee6c88d94894b5f2625d83c4f028" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 43167 } } peers { permanent_uuid: "6f041c2e4531499f8d180c6d49e8e4ae" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 38165 } }
I20260928 06:30:12.909650 2686 ts_tablet_manager.cc:1428] T 72997cc4ce734d5d80f29969c5f9ed02 P 6f041c2e4531499f8d180c6d49e8e4ae: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260928 06:30:12.909881 2532 heartbeater.cc:503] Master 127.22.225.190:45141 was elected leader, sending a full tablet report...
I20260928 06:30:12.913000 2681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:30:12.884028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.131:43167 (local address 127.0.0.1:60778)
0928 06:30:12.886056 (+ 2028us) negotiation.cc:107] Waiting for socket to connect
0928 06:30:12.886106 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 06:30:12.886190 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:30:12.899925 (+ 13735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:30:12.899926 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:30:12.899928 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:30:12.899971 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:30:12.899972 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:30:12.907709 (+ 7737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:30:12.907711 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:30:12.909151 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:30:12.909155 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:30:12.912761 (+ 3606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:30:12.912762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:30:12.912834 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:30:12.912840 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 06:30:12.912869 (+ 29us) client_negotiation.cc:240] Negotiation successful
0928 06:30:12.912917 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1964,"thread_start_us":137,"threads_started":1}
I20260928 06:30:12.913554 2682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:30:12.884759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60778 (local address 127.22.225.131:43167)
0928 06:30:12.894409 (+ 9650us) server_negotiation.cc:206] Beginning negotiation
0928 06:30:12.894414 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:30:12.894426 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:30:12.894496 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:30:12.894498 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:30:12.894515 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:30:12.894597 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:30:12.905265 (+ 10668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:30:12.907631 (+ 2366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:30:12.911530 (+ 3899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:30:12.912665 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:30:12.913335 (+ 670us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:30:12.913410 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 06:30:12.913437 (+ 27us) server_negotiation.cc:299] Negotiation successful
0928 06:30:12.913481 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9536,"thread_start_us":97,"threads_started":1}
I20260928 06:30:12.916007 2601 tablet_service.cc:1467] Processing CreateTablet for tablet 72997cc4ce734d5d80f29969c5f9ed02 (DEFAULT_TABLE table=test-table [id=712316e49d4d42cf930b8bd6f1ef1df0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:30:12.916412 2601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72997cc4ce734d5d80f29969c5f9ed02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:30:12.916414 2688 tablet_bootstrap.cc:492] T 72997cc4ce734d5d80f29969c5f9ed02 P 775328fa003c413290f840c989024f57: Bootstrap starting.
I20260928 06:30:12.917805 2688 tablet_bootstrap.cc:654] T 72997cc4ce734d5d80f29969c5f9ed02 P 775328fa003c413290f840c989024f57: Neither blocks nor log segments found. Creating new log.
I20260928 06:30:12.919727 2688 log.cc:824] T 72997cc4ce734d5d80f29969c5f9ed02 P 775328fa003c413290f840c989024f57: Log is configured to *not* fsync() on all Append() calls
I20260928 06:30:12.921234 2690 tablet_bootstrap.cc:492] T 72997cc4ce734d5d80f29969c5f9ed02 P 1011ee6c88d94894b5f2625d83c4f028: Bootstrap starting.
I20260928 06:30:12.922619 2690 tablet_bootstrap.cc:654] T 72997cc4ce734d5d80f29969c5f9ed02 P 1011ee6c88d94894b5f2625d83c4f028: Neither blocks nor log segments found. Creating new log.
I20260928 06:30:12.923225 2690 log.cc:824] T 72997cc4ce734d5d80f29969c5f9ed02 P 1011ee6c88d94894b5f2625d83c4f028: Log is configured to *not* fsync() on all Append() calls
I20260928 06:30:12.924268 2688 tablet_bootstrap.cc:492] T 72997cc4ce734d5d80f29969c5f9ed02 P 775328fa003c413290f840c989024f57: No bootstrap required, opened a new log
I20260928 06:30:12.924351 2688 ts_tablet_manager.cc:1397] T 72997cc4ce734d5d80f29969c5f9ed02 P 775328fa003c413290f840c989024f57: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20260928 06:30:12.929342 2688 raft_consensus.cc:348] T 72997cc4ce734d5d80f29969c5f9ed02 P 775328fa003c413290f840c989024f57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "775328fa003c413290f840c989024f57" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 41669 } } peers { permanent_uuid: "1011ee6c88d94894b5f2625d83c4f028" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 43167 } } peers { permanent_uuid: "6f041c2e4531499f8d180c6d49e8e4ae" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 38165 } }
I20260928 06:30:12.929539 2688 raft_consensus.cc:374] T 72997cc4ce734d5d80f29969c5f9ed02 P 775328fa003c413290f840c989024f57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:30:12.929587 2688 raft_consensus.cc:729] T 72997cc4ce734d5d80f29969c5f9ed02 P 775328fa003c413290f840c989024f57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 775328fa003c413290f840c989024f57, State: Initialized, Role: FOLLOWER
I20260928 06:30:12.930450 2690 tablet_bootstrap.cc:492] T 72997cc4ce734d5d80f29969c5f9ed02 P 1011ee6c88d94894b5f2625d83c4f028: No bootstrap required, opened a new log
I20260928 06:30:12.930532 2690 ts_tablet_manager.cc:1397] T 72997cc4ce734d5d80f29969c5f9ed02 P 1011ee6c88d94894b5f2625d83c4f028: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260928 06:30:12.934578 2690 raft_consensus.cc:348] T 72997cc4ce734d5d80f29969c5f9ed02 P 1011ee6c88d94894b5f2625d83c4f028 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "775328fa003c413290f840c989024f57" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 41669 } } peers { permanent_uuid: "1011ee6c88d94894b5f2625d83c4f028" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 43167 } } peers { permanent_uuid: "6f041c2e4531499f8d180c6d49e8e4ae" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 38165 } }
I20260928 06:30:12.934839 2690 raft_consensus.cc:374] T 72997cc4ce734d5d80f29969c5f9ed02 P 1011ee6c88d94894b5f2625d83c4f028 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:30:12.934891 2690 raft_consensus.cc:729] T 72997cc4ce734d5d80f29969c5f9ed02 P 1011ee6c88d94894b5f2625d83c4f028 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1011ee6c88d94894b5f2625d83c4f028, State: Initialized, Role: FOLLOWER
I20260928 06:30:12.935058 2690 consensus_queue.cc:260] T 72997cc4ce734d5d80f29969c5f9ed02 P 1011ee6c88d94894b5f2625d83c4f028 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "775328fa003c413290f840c989024f57" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 41669 } } peers { permanent_uuid: "1011ee6c88d94894b5f2625d83c4f028" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 43167 } } peers { permanent_uuid: "6f041c2e4531499f8d180c6d49e8e4ae" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 38165 } }
I20260928 06:30:12.935457 2690 ts_tablet_manager.cc:1428] T 72997cc4ce734d5d80f29969c5f9ed02 P 1011ee6c88d94894b5f2625d83c4f028: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 06:30:12.935606 2667 heartbeater.cc:503] Master 127.22.225.190:45141 was elected leader, sending a full tablet report...
I20260928 06:30:12.936767 2688 consensus_queue.cc:260] T 72997cc4ce734d5d80f29969c5f9ed02 P 775328fa003c413290f840c989024f57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "775328fa003c413290f840c989024f57" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 41669 } } peers { permanent_uuid: "1011ee6c88d94894b5f2625d83c4f028" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 43167 } } peers { permanent_uuid: "6f041c2e4531499f8d180c6d49e8e4ae" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 38165 } }
I20260928 06:30:12.937196 2688 ts_tablet_manager.cc:1428] T 72997cc4ce734d5d80f29969c5f9ed02 P 775328fa003c413290f840c989024f57: Time spent starting tablet: real 0.013s user 0.003s sys 0.003s
I20260928 06:30:12.937332 2395 heartbeater.cc:503] Master 127.22.225.190:45141 was elected leader, sending a full tablet report...
W20260928 06:30:12.966005 2396 tablet.cc:2369] T 72997cc4ce734d5d80f29969c5f9ed02 P 775328fa003c413290f840c989024f57: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:30:13.012102 2533 tablet.cc:2369] T 72997cc4ce734d5d80f29969c5f9ed02 P 6f041c2e4531499f8d180c6d49e8e4ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:30:13.067029 2668 tablet.cc:2369] T 72997cc4ce734d5d80f29969c5f9ed02 P 1011ee6c88d94894b5f2625d83c4f028: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 06:30:13 dist-test-slave-d59m krb5kdc[1705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577012, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.190@KRBTEST.COM for HTTP/127.22.225.148@KRBTEST.COM
I20260928 06:30:13.120070 2689 raft_consensus.cc:482] T 72997cc4ce734d5d80f29969c5f9ed02 P 6f041c2e4531499f8d180c6d49e8e4ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:30:13.120219 2689 raft_consensus.cc:504] T 72997cc4ce734d5d80f29969c5f9ed02 P 6f041c2e4531499f8d180c6d49e8e4ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "775328fa003c413290f840c989024f57" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 41669 } } peers { permanent_uuid: "1011ee6c88d94894b5f2625d83c4f028" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 43167 } } peers { permanent_uuid: "6f041c2e4531499f8d180c6d49e8e4ae" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 38165 } }
I20260928 06:30:13.120724 2689 leader_election.cc:288] T 72997cc4ce734d5d80f29969c5f9ed02 P 6f041c2e4531499f8d180c6d49e8e4ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 775328fa003c413290f840c989024f57 (127.22.225.129:41669), 1011ee6c88d94894b5f2625d83c4f028 (127.22.225.131:43167)
I20260928 06:30:13.125722 2685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:30:13.121366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:56949 (local address 127.22.225.129:41669)
0928 06:30:13.121418 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 06:30:13.121420 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:30:13.121811 (+ 391us) server_negotiation.cc:407] Connection header received
0928 06:30:13.121900 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:30:13.121901 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:30:13.121912 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:30:13.121966 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:30:13.122155 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:30:13.122806 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:30:13.124677 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:30:13.125235 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:30:13.125315 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:30:13.125324 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:30:13.125532 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 06:30:13.125593 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 06:30:13.126192 2695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:30:13.121039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.129:41669 (local address 127.22.225.130:56949)
0928 06:30:13.121709 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0928 06:30:13.121721 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:30:13.121820 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:30:13.121969 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:30:13.121970 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:30:13.121972 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:30:13.122075 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:30:13.122077 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:30:13.122896 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:30:13.122897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:30:13.124464 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:30:13.124468 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:30:13.125318 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:30:13.125320 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:30:13.125410 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:30:13.125417 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 06:30:13.126054 (+ 637us) client_negotiation.cc:240] Negotiation successful
0928 06:30:13.126113 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":70,"threads_started":1}
I20260928 06:30:13.126663 2349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72997cc4ce734d5d80f29969c5f9ed02" candidate_uuid: "6f041c2e4531499f8d180c6d49e8e4ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "775328fa003c413290f840c989024f57" is_pre_election: true
I20260928 06:30:13.126847 2349 raft_consensus.cc:2454] T 72997cc4ce734d5d80f29969c5f9ed02 P 775328fa003c413290f840c989024f57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f041c2e4531499f8d180c6d49e8e4ae in term 0.
I20260928 06:30:13.127084 2420 leader_election.cc:302] T 72997cc4ce734d5d80f29969c5f9ed02 P 6f041c2e4531499f8d180c6d49e8e4ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f041c2e4531499f8d180c6d49e8e4ae, 775328fa003c413290f840c989024f57; no voters:
I20260928 06:30:13.127214 2689 raft_consensus.cc:2781] T 72997cc4ce734d5d80f29969c5f9ed02 P 6f041c2e4531499f8d180c6d49e8e4ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:30:13.127252 2689 raft_consensus.cc:482] T 72997cc4ce734d5d80f29969c5f9ed02 P 6f041c2e4531499f8d180c6d49e8e4ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:30:13.127274 2689 raft_consensus.cc:3037] T 72997cc4ce734d5d80f29969c5f9ed02 P 6f041c2e4531499f8d180c6d49e8e4ae [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:30:13.128189 2689 raft_consensus.cc:504] T 72997cc4ce734d5d80f29969c5f9ed02 P 6f041c2e4531499f8d180c6d49e8e4ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "775328fa003c413290f840c989024f57" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 41669 } } peers { permanent_uuid: "1011ee6c88d94894b5f2625d83c4f028" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 43167 } } peers { permanent_uuid: "6f041c2e4531499f8d180c6d49e8e4ae" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 38165 } }
I20260928 06:30:13.128319 2689 leader_election.cc:288] T 72997cc4ce734d5d80f29969c5f9ed02 P 6f041c2e4531499f8d180c6d49e8e4ae [CANDIDATE]: Term 1 election: Requested vote from peers 775328fa003c413290f840c989024f57 (127.22.225.129:41669), 1011ee6c88d94894b5f2625d83c4f028 (127.22.225.131:43167)
I20260928 06:30:13.128683 2349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72997cc4ce734d5d80f29969c5f9ed02" candidate_uuid: "6f041c2e4531499f8d180c6d49e8e4ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "775328fa003c413290f840c989024f57"
I20260928 06:30:13.128767 2349 raft_consensus.cc:3037] T 72997cc4ce734d5d80f29969c5f9ed02 P 775328fa003c413290f840c989024f57 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:30:13.129609 2349 raft_consensus.cc:2454] T 72997cc4ce734d5d80f29969c5f9ed02 P 775328fa003c413290f840c989024f57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f041c2e4531499f8d180c6d49e8e4ae in term 1.
I20260928 06:30:13.129794 2420 leader_election.cc:302] T 72997cc4ce734d5d80f29969c5f9ed02 P 6f041c2e4531499f8d180c6d49e8e4ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f041c2e4531499f8d180c6d49e8e4ae, 775328fa003c413290f840c989024f57; no voters:
I20260928 06:30:13.129886 2689 raft_consensus.cc:2781] T 72997cc4ce734d5d80f29969c5f9ed02 P 6f041c2e4531499f8d180c6d49e8e4ae [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:30:13.130053 2689 raft_consensus.cc:686] T 72997cc4ce734d5d80f29969c5f9ed02 P 6f041c2e4531499f8d180c6d49e8e4ae [term 1 LEADER]: Becoming Leader. State: Replica: 6f041c2e4531499f8d180c6d49e8e4ae, State: Running, Role: LEADER
I20260928 06:30:13.130151 2689 consensus_queue.cc:237] T 72997cc4ce734d5d80f29969c5f9ed02 P 6f041c2e4531499f8d180c6d49e8e4ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "775328fa003c413290f840c989024f57" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 41669 } } peers { permanent_uuid: "1011ee6c88d94894b5f2625d83c4f028" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 43167 } } peers { permanent_uuid: "6f041c2e4531499f8d180c6d49e8e4ae" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 38165 } }
I20260928 06:30:13.130959 2182 catalog_manager.cc:5495] T 72997cc4ce734d5d80f29969c5f9ed02 P 6f041c2e4531499f8d180c6d49e8e4ae reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f041c2e4531499f8d180c6d49e8e4ae (127.22.225.130). New cstate: current_term: 1 leader_uuid: "6f041c2e4531499f8d180c6d49e8e4ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "775328fa003c413290f840c989024f57" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 41669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1011ee6c88d94894b5f2625d83c4f028" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 43167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f041c2e4531499f8d180c6d49e8e4ae" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 38165 } health_report { overall_health: HEALTHY } } }
I20260928 06:30:13.134203 2682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:30:13.121656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:43741 (local address 127.22.225.131:43167)
0928 06:30:13.121710 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 06:30:13.121712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:30:13.125443 (+ 3731us) server_negotiation.cc:407] Connection header received
0928 06:30:13.125543 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:30:13.125545 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:30:13.125554 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:30:13.125582 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:30:13.129419 (+ 3837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:30:13.130124 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:30:13.132763 (+ 2639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:30:13.133296 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:30:13.133348 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:30:13.133358 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 06:30:13.134086 (+ 728us) server_negotiation.cc:299] Negotiation successful
0928 06:30:13.134135 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 06:30:13.134575 2696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:30:13.121464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.131:43167 (local address 127.22.225.130:43741)
0928 06:30:13.125382 (+ 3918us) negotiation.cc:107] Waiting for socket to connect
0928 06:30:13.125393 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:30:13.125474 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:30:13.125986 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:30:13.125987 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:30:13.125989 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:30:13.126077 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:30:13.126078 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:30:13.130237 (+ 4159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:30:13.130238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:30:13.131306 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:30:13.131309 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:30:13.133421 (+ 2112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:30:13.133422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:30:13.133495 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:30:13.133499 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:30:13.134430 (+ 931us) client_negotiation.cc:240] Negotiation successful
0928 06:30:13.134478 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3900,"thread_start_us":46,"threads_started":1}
I20260928 06:30:13.135342 2621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72997cc4ce734d5d80f29969c5f9ed02" candidate_uuid: "6f041c2e4531499f8d180c6d49e8e4ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1011ee6c88d94894b5f2625d83c4f028" is_pre_election: true
I20260928 06:30:13.135502 2621 raft_consensus.cc:2454] T 72997cc4ce734d5d80f29969c5f9ed02 P 1011ee6c88d94894b5f2625d83c4f028 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f041c2e4531499f8d180c6d49e8e4ae in term 0.
I20260928 06:30:13.135617 2620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72997cc4ce734d5d80f29969c5f9ed02" candidate_uuid: "6f041c2e4531499f8d180c6d49e8e4ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1011ee6c88d94894b5f2625d83c4f028"
I20260928 06:30:13.135675 2620 raft_consensus.cc:3037] T 72997cc4ce734d5d80f29969c5f9ed02 P 1011ee6c88d94894b5f2625d83c4f028 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:30:13.136525 2620 raft_consensus.cc:2454] T 72997cc4ce734d5d80f29969c5f9ed02 P 1011ee6c88d94894b5f2625d83c4f028 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f041c2e4531499f8d180c6d49e8e4ae in term 1.
I20260928 06:30:13.177423 2683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:30:13.174020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51388 (local address 127.22.225.130:38165)
0928 06:30:13.174067 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 06:30:13.174069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:30:13.174098 (+ 29us) server_negotiation.cc:407] Connection header received
0928 06:30:13.174204 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:30:13.174205 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:30:13.174228 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:30:13.174321 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:30:13.174436 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:30:13.175366 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:30:13.176471 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:30:13.176791 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:30:13.176873 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:30:13.177020 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:30:13.177163 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:30:13.177269 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 06:30:13.177288 (+ 19us) server_negotiation.cc:299] Negotiation successful
0928 06:30:13.177310 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 06:30:13.209172 2349 raft_consensus.cc:1261] T 72997cc4ce734d5d80f29969c5f9ed02 P 775328fa003c413290f840c989024f57 [term 1 FOLLOWER]: Refusing update from remote peer 6f041c2e4531499f8d180c6d49e8e4ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:30:13.209172 2620 raft_consensus.cc:1261] T 72997cc4ce734d5d80f29969c5f9ed02 P 1011ee6c88d94894b5f2625d83c4f028 [term 1 FOLLOWER]: Refusing update from remote peer 6f041c2e4531499f8d180c6d49e8e4ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:30:13.209754 2697 consensus_queue.cc:1035] T 72997cc4ce734d5d80f29969c5f9ed02 P 6f041c2e4531499f8d180c6d49e8e4ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "1011ee6c88d94894b5f2625d83c4f028" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 43167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:30:13.209923 2689 consensus_queue.cc:1035] T 72997cc4ce734d5d80f29969c5f9ed02 P 6f041c2e4531499f8d180c6d49e8e4ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "775328fa003c413290f840c989024f57" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 41669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:30:13.214150 2700 mvcc.cc:205] Tried to move back new op lower bound from 7334203446100967424 to 7334203445781663744. Current Snapshot: MvccSnapshot[applied={T|T < 7334203446100967424}]
I20260928 06:30:13.220919 2701 mvcc.cc:205] Tried to move back new op lower bound from 7334203446100967424 to 7334203445781663744. Current Snapshot: MvccSnapshot[applied={T|T < 7334203446100967424}]
I20260928 06:30:13.223817 2182 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53486:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:30:13.223942 2182 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53486:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:30:13.224750 2182 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ae4a5f0e447c475584d6f20c69a53d47: Sending DeleteTablet for 3 replicas of tablet 72997cc4ce734d5d80f29969c5f9ed02
I20260928 06:30:13.227957 2466 tablet_service.cc:1514] Processing DeleteTablet for tablet 72997cc4ce734d5d80f29969c5f9ed02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:30:13 UTC) from {username='kudu', principal='kudu/127.22.225.190@KRBTEST.COM'} at 127.0.0.1:51386
I20260928 06:30:13.228190 2329 tablet_service.cc:1514] Processing DeleteTablet for tablet 72997cc4ce734d5d80f29969c5f9ed02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:30:13 UTC) from {username='kudu', principal='kudu/127.22.225.190@KRBTEST.COM'} at 127.0.0.1:54242
I20260928 06:30:13.228514 2601 tablet_service.cc:1514] Processing DeleteTablet for tablet 72997cc4ce734d5d80f29969c5f9ed02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:30:13 UTC) from {username='kudu', principal='kudu/127.22.225.190@KRBTEST.COM'} at 127.0.0.1:60778
I20260928 06:30:13.229050 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 2264
I20260928 06:30:13.229058 2702 mvcc.cc:205] Tried to move back new op lower bound from 7334203446100967424 to 7334203445781663744. Current Snapshot: MvccSnapshot[applied={T|T < 7334203446100967424}]
I20260928 06:30:13.236773 2711 tablet_replica.cc:330] T 72997cc4ce734d5d80f29969c5f9ed02 P 1011ee6c88d94894b5f2625d83c4f028: stopping tablet replica
W20260928 06:30:13.237573 2167 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.225.129:41669 (error 108)
W20260928 06:30:13.238193 2681 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:30:13.238026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.129:41669 (local address 127.0.0.1:54244)
0928 06:30:13.238073 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 06:30:13.238085 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:30:13.238140 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.129:41669: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":23}
W20260928 06:30:13.238408 2167 catalog_manager.cc:4584] TS 775328fa003c413290f840c989024f57 (127.22.225.129:41669): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 72997cc4ce734d5d80f29969c5f9ed02: Network error: Client connection negotiation failed: client connection to 127.22.225.129:41669: BlockingWrite error: write error: Connection reset by peer (error 104)
I20260928 06:30:13.238778 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 2401
I20260928 06:30:13.239513 2711 raft_consensus.cc:2229] T 72997cc4ce734d5d80f29969c5f9ed02 P 1011ee6c88d94894b5f2625d83c4f028 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:30:13.239646 2711 raft_consensus.cc:2258] T 72997cc4ce734d5d80f29969c5f9ed02 P 1011ee6c88d94894b5f2625d83c4f028 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:30:13.242576 2711 ts_tablet_manager.cc:1905] T 72997cc4ce734d5d80f29969c5f9ed02 P 1011ee6c88d94894b5f2625d83c4f028: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:30:13.247108 2711 ts_tablet_manager.cc:1918] T 72997cc4ce734d5d80f29969c5f9ed02 P 1011ee6c88d94894b5f2625d83c4f028: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:30:13.248674 2711 log.cc:1196] T 72997cc4ce734d5d80f29969c5f9ed02 P 1011ee6c88d94894b5f2625d83c4f028: Deleting WAL directory at /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576899143097-23430-0/minicluster-data/ts-2/wal/wals/72997cc4ce734d5d80f29969c5f9ed02
I20260928 06:30:13.249177 2711 ts_tablet_manager.cc:1939] T 72997cc4ce734d5d80f29969c5f9ed02 P 1011ee6c88d94894b5f2625d83c4f028: Deleting consensus metadata
I20260928 06:30:13.249848 2166 catalog_manager.cc:4841] TS 1011ee6c88d94894b5f2625d83c4f028 (127.22.225.131:43167): tablet 72997cc4ce734d5d80f29969c5f9ed02 (table test-table [id=712316e49d4d42cf930b8bd6f1ef1df0]) successfully deleted
I20260928 06:30:13.264758 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 2538
I20260928 06:30:13.266388 2681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:30:13.266225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.130:38165 (local address 127.0.0.1:51390)
0928 06:30:13.266290 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 06:30:13.266346 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.130:38165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 06:30:13.271435 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 2150
I20260928 06:30:13.284718 23430 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 06:30:13.629155 23430 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 06:30:13.629272 23430 mini_ranger.cc:66] Stopping Ranger...
I20260928 06:30:13.973321 23430 mini_ranger.cc:68] Stopped Ranger
2026-09-28 06:30:13.973 UTC [1735] LOG: received smart shutdown request
2026-09-28 06:30:13.977 UTC [1735] LOG: background worker "logical replication launcher" (PID 1744) exited with exit code 1
2026-09-28 06:30:13.978 UTC [1739] LOG: shutting down
2026-09-28 06:30:14.088 UTC [1735] LOG: database system is shut down
2026-09-28T06:30:14Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (57371 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:30:14 dist-test-slave-d59m krb5kdc[2722](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:30:14 dist-test-slave-d59m krb5kdc[2722](info): set up 2 sockets
Sep 28 06:30:14 dist-test-slave-d59m krb5kdc[2722](info): commencing operation
krb5kdc: starting...
W20260928 06:30:15.253981 23430 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Sep 28 06:30:15 dist-test-slave-d59m krb5kdc[2722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:30:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:30:15Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.22.225.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.22.225.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.22.225.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/krb5kdc/rangeradmin_127.22.225.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.22.225.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/krb5kdc/rangeradmin_127.22.225.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.22.225.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.22.225.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.22.225.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/krb5kdc/rangerlookup_127.22.225.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.22.225.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/krb5kdc/rangerlookup_127.22.225.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.22.225.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.225.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.225.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/krb5kdc/HTTP_127.22.225.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.22.225.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/krb5kdc/HTTP_127.22.225.148@KRBTEST.COM.keytab.
I20260928 06:30:15.428480 23430 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskcCVV2M/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/postgres -l logfile start
2026-09-28 06:30:18.028 UTC [2752] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 06:30:18.028 UTC [2752] LOG: listening on IPv4 address "127.22.225.148", port 57091
2026-09-28 06:30:18.032 UTC [2752] LOG: listening on Unix socket "/tmp/.s.PGSQL.57091"
2026-09-28 06:30:18.066 UTC [2755] LOG: database system was shut down at 2026-09-28 06:30:16 UTC
2026-09-28 06:30:18.070 UTC [2752] LOG: database system is ready to accept connections
I20260928 06:30:19.032752 23430 mini_postgres.cc:96] Postgres bound to 57091
2026-09-28 06:30:19.038 UTC [2763] FATAL: database "slave" does not exist
127.22.225.148:57091 - accepting connections
I20260928 06:30:19.039335 23430 mini_ranger.cc:161] Starting Ranger...
I20260928 06:30:19.054425 23430 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 06:30:19.545094 23430 mini_ranger.cc:87] Created ranger Postgres database
I20260928 06:30:19.545181 23430 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ranger-admin
2026-09-28 06:30:19,850 [I] DB FLAVOR :POSTGRES
2026-09-28 06:30:19,851 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 06:30:19,851 [I] Checking connection..
2026-09-28 06:30:19,851 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcCVV2M/build/release/bin/postgresql.jar:/tmp/dist-test-taskcCVV2M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.225.148:57091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 06:30:20,143 [I] Checking connection passed.
2026-09-28 06:30:20,143 [I] --------- Verifying version history table ---------
2026-09-28 06:30:20,143 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcCVV2M/build/release/bin/postgresql.jar:/tmp/dist-test-taskcCVV2M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.225.148:57091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 06:30:20,421 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 06:30:20,422 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcCVV2M/build/release/bin/postgresql.jar:/tmp/dist-test-taskcCVV2M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.225.148:57091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 06:30:20,732 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 06:30:20,732 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 06:30:20,732 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcCVV2M/build/release/bin/postgresql.jar:/tmp/dist-test-taskcCVV2M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.225.148:57091/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcCVV2M/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 06:30:21.014 UTC [2827] WARNING: there is no transaction in progress
2026-09-28 06:30:21,021 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 06:30:21,021 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcCVV2M/build/release/bin/postgresql.jar:/tmp/dist-test-taskcCVV2M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.225.148:57091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 06:30:21,303 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 06:30:21,303 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 06:30:21,303 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcCVV2M/build/release/bin/postgresql.jar:/tmp/dist-test-taskcCVV2M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.225.148:57091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 06:30:21,560 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcCVV2M/build/release/bin/postgresql.jar:/tmp/dist-test-taskcCVV2M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.225.148:57091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 06:30:21,832 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcCVV2M/build/release/bin/postgresql.jar:/tmp/dist-test-taskcCVV2M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.225.148:57091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-d59m.c.gcp-upstream.internal','N') ;"
2026-09-28 06:30:22,114 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcCVV2M/build/release/bin/postgresql.jar:/tmp/dist-test-taskcCVV2M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.225.148:57091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 06:30:22,387 [I] Table x_portal_user does not exist in database ranger
2026-09-28 06:30:22,388 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcCVV2M/build/release/bin/postgresql.jar:/tmp/dist-test-taskcCVV2M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.225.148:57091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 06:30:22,694 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 06:30:22,694 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcCVV2M/build/release/bin/postgresql.jar:/tmp/dist-test-taskcCVV2M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.225.148:57091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 06:30:22,992 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcCVV2M/build/release/bin/postgresql.jar:/tmp/dist-test-taskcCVV2M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.225.148:57091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 06:30:23,280 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 06:30:23,281 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcCVV2M/build/release/bin/postgresql.jar:/tmp/dist-test-taskcCVV2M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.225.148:57091/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcCVV2M/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 06:30:24.314 UTC [2935] WARNING: there is no transaction in progress
2026-09-28 06:30:24.334 UTC [2935] WARNING: there is no transaction in progress
2026-09-28 06:30:24.354 UTC [2935] WARNING: there is no transaction in progress
2026-09-28 06:30:24.370 UTC [2935] WARNING: there is no transaction in progress
2026-09-28 06:30:24.390 UTC [2935] WARNING: there is no transaction in progress
2026-09-28 06:30:24.415 UTC [2935] WARNING: there is no transaction in progress
2026-09-28 06:30:24.481 UTC [2935] WARNING: there is no transaction in progress
2026-09-28 06:30:24.488 UTC [2935] WARNING: there is no transaction in progress
2026-09-28 06:30:24.504 UTC [2935] WARNING: there is no transaction in progress
2026-09-28 06:30:24.517 UTC [2935] WARNING: there is no transaction in progress
2026-09-28 06:30:24.529 UTC [2935] WARNING: there is no transaction in progress
2026-09-28 06:30:24.547 UTC [2935] WARNING: there is no transaction in progress
2026-09-28 06:30:24.561 UTC [2935] WARNING: there is no transaction in progress
2026-09-28 06:30:24.567 UTC [2935] WARNING: there is no transaction in progress
2026-09-28 06:30:24.574 UTC [2935] WARNING: there is no transaction in progress
2026-09-28 06:30:24.581 UTC [2935] WARNING: there is no transaction in progress
2026-09-28 06:30:25,252 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 06:30:25,252 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcCVV2M/build/release/bin/postgresql.jar:/tmp/dist-test-taskcCVV2M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.225.148:57091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 06:30:25,504 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 06:30:25,505 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcCVV2M/build/release/bin/postgresql.jar:/tmp/dist-test-taskcCVV2M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.225.148:57091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 06:30:25,824 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 06:30:25,825 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcCVV2M/build/release/bin/postgresql.jar:/tmp/dist-test-taskcCVV2M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.225.148:57091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 06:30:26,118 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 06:30:26,118 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcCVV2M/build/release/bin/postgresql.jar:/tmp/dist-test-taskcCVV2M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.225.148:57091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 06:30:26,387 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcCVV2M/build/release/bin/postgresql.jar:/tmp/dist-test-taskcCVV2M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.225.148:57091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 06:30:26,662 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcCVV2M/build/release/bin/postgresql.jar:/tmp/dist-test-taskcCVV2M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.225.148:57091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-d59m.c.gcp-upstream.internal';"
2026-09-28 06:30:26,945 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 06:30:26,946 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcCVV2M/build/release/bin/postgresql.jar:/tmp/dist-test-taskcCVV2M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.225.148:57091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 06:30:27,214 [I] DB_PATCHES have already been applied
I20260928 06:30:27.219883 23430 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ranger-admin:/tmp/dist-test-taskcCVV2M/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskcCVV2M/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskcCVV2M/build/release/bin/postgresql.jar:/tmp/dist-test-taskcCVV2M/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 06:30:27.219971 23430 mini_ranger.cc:193] Using host: 127.22.225.148
I20260928 06:30:27.223182 23430 mini_ranger.cc:227] Ranger admin URL: http://127.22.225.148:38615
Sep 28, 2026 6:30:27 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 6:30:27 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 6:30:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 6:30:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 6:30:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 6:30:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 6:30:28 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 06:30:28.399979 23430 mini_ranger.cc:160] Time spent starting Ranger: real 9.361s user 0.000s sys 0.009s
Sep 28, 2026 6:30:51 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 6:30:51 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 28, 2026 6:30:51 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 6:30:51 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 6:30:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 6:30:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 6:30:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 6:30:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 6:30:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 6:30:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 6:30:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 6:30:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 6:30:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 6:30:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 6:30:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 6:30:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 6:30:52 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 6:30:53 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 06:30:53.763828 23430 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.22.225.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.22.225.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.22.225.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/krb5kdc/rangerkms_127.22.225.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.22.225.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/krb5kdc/rangerkms_127.22.225.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.22.225.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.22.225.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.225.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/krb5kdc/HTTP_127.22.225.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.22.225.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/krb5kdc/HTTP_127.22.225.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 06:30:53 dist-test-slave-d59m krb5kdc[2722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577053, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 06:30:53.865901 23430 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 06:30:53.888571 23430 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 06:30:54.862488 23430 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 06:30:54.862573 23430 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ranger-kms
I20260928 06:30:54.862592 23430 mini_ranger_kms.cc:227] Using postgres at 127.22.225.148:57091
2026-09-28 06:30:55,066 [I] DB FLAVOR :POSTGRES
2026-09-28 06:30:55,066 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 06:30:55,066 [I] Checking connection
2026-09-28 06:30:55,066 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcCVV2M/build/release/bin/postgresql.jar:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.225.148:57091/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 06:30:55,399 [I] connection success
2026-09-28 06:30:55,400 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 06:30:55,400 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 06:30:55,400 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcCVV2M/build/release/bin/postgresql.jar:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.225.148:57091/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 06:30:55,716 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 06:30:55,717 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 06:30:55,717 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 06:30:55,717 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcCVV2M/build/release/bin/postgresql.jar:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.225.148:57091/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 06:30:56,057 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 06:30:56.701766 23430 mini_ranger_kms.cc:310] Created kms service
I20260928 06:30:56.940790 23430 mini_ranger_kms.cc:326] Created kudu user
I20260928 06:30:57.005446 23430 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 06:30:57 dist-test-slave-d59m krb5kdc[2722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577053, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.22.225.148@KRBTEST.COM
I20260928 06:30:57.550114 23430 mini_ranger_kms.cc:384] Added ranger policy
I20260928 06:30:57.550321 23430 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ranger-kms:/tmp/dist-test-taskcCVV2M/build/release/bin/postgresql.jar:/tmp/dist-test-taskcCVV2M/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskcCVV2M/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 06:30:57.550344 23430 mini_ranger_kms.cc:240] Using host: 127.22.225.148
I20260928 06:30:57.554536 23430 mini_ranger_kms.cc:276] Ranger KMS PID: 3116
I20260928 06:30:57.554620 23430 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.22.225.148:51731
Sep 28, 2026 6:30:59 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 6:30:59 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 6:30:59 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 6:30:59 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 6:30:59 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 6:30:59 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 06:30:59.933086 23430 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.067s user 0.007s sys 0.173s
I20260928 06:30:59.933197 23430 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 06:30:59.933243 23430 mini_ranger_kms.cc:401] 127.22.225.148:51731/kms/v1/keys
Sep 28 06:31:00 dist-test-slave-d59m krb5kdc[2722](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790577060, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.225.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 06:31:01 dist-test-slave-d59m krb5kdc[2722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577060, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.225.148@KRBTEST.COM for HTTP/127.22.225.148@KRBTEST.COM
Sep 28 06:31:01 dist-test-slave-d59m krb5kdc[2722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577060, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.225.148@KRBTEST.COM for HTTP/127.22.225.148@KRBTEST.COM
Sep 28 06:31:03 dist-test-slave-d59m krb5kdc[2722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.22.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab.
Entry for principal kudu/127.22.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab.
I20260928 06:31:03.622189 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:43353
--webserver_interface=127.22.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab
--principal=kudu/127.22.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:44983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.225.190:43353
--ranger_config_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.225.148:51731/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:31:03.724056 3166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:03.724308 3166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:03.724345 3166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:03.725649 3166 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:31:03.725723 3166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:03.725746 3166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:03.725775 3166 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:31:03.725806 3166 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:31:03.727422 3166 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:44983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/master-0/wal
--ranger_kms_url=127.22.225.148:51731/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.22.225.190:43353
--ranger_config_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab
--principal=kudu/127.22.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:43353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:31:03.727712 3166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:03.727954 3166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:03.733984 3171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:03.733960 3172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:03.734213 3166 server_base.cc:1034] running on GCE node
W20260928 06:31:03.733984 3174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:03.734555 3166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:03.734856 3166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:03.736052 3166 hybrid_clock.cc:648] HybridClock initialized: now 1790577063736029 us; error 39 us; skew 500 ppm
Sep 28 06:31:03 dist-test-slave-d59m krb5kdc[2722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577063, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:03.742239 3166 init.cc:377] Logged in from keytab as kudu/127.22.225.190@KRBTEST.COM (short username kudu)
I20260928 06:31:03.742825 3166 webserver.cc:460] Webserver started at http://127.22.225.190:40811/ using document root <none> and password file <none>
I20260928 06:31:03.743088 3166 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:03.743152 3166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:03.743311 3166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 06:31:03 dist-test-slave-d59m krb5kdc[2722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577063, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.190@KRBTEST.COM for HTTP/127.22.225.148@KRBTEST.COM
I20260928 06:31:03.805984 3166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/master-0/data/instance:
uuid: "fb7bf101bf764f27960d5a5ad1825a73"
format_stamp: "Formatted at 2026-09-28 06:31:03 on dist-test-slave-d59m"
server_key: "31b0310b4429e723825d62e0724e67a6"
server_key_iv: "84a1aeb7293508815b744c8019719843"
server_key_version: "kuduclusterkey@0"
I20260928 06:31:03.806404 3166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/master-0/wal/instance:
uuid: "fb7bf101bf764f27960d5a5ad1825a73"
format_stamp: "Formatted at 2026-09-28 06:31:03 on dist-test-slave-d59m"
server_key: "31b0310b4429e723825d62e0724e67a6"
server_key_iv: "84a1aeb7293508815b744c8019719843"
server_key_version: "kuduclusterkey@0"
I20260928 06:31:03.807917 3166 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:31:03.823570 3181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:03.823815 3166 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:31:03.823925 3166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/master-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/master-0/wal
uuid: "fb7bf101bf764f27960d5a5ad1825a73"
format_stamp: "Formatted at 2026-09-28 06:31:03 on dist-test-slave-d59m"
server_key: "31b0310b4429e723825d62e0724e67a6"
server_key_iv: "84a1aeb7293508815b744c8019719843"
server_key_version: "kuduclusterkey@0"
I20260928 06:31:03.824033 3166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:03.834350 3166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:03.836422 3166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:03.836607 3166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:03.841696 3166 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.190:43353
I20260928 06:31:03.841727 3233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.190:43353 every 8 connection(s)
I20260928 06:31:03.842159 3166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
I20260928 06:31:03.842876 3234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:03.845248 3234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb7bf101bf764f27960d5a5ad1825a73: Bootstrap starting.
I20260928 06:31:03.845852 3234 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb7bf101bf764f27960d5a5ad1825a73: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:03.846138 3234 log.cc:824] T 00000000000000000000000000000000 P fb7bf101bf764f27960d5a5ad1825a73: Log is configured to *not* fsync() on all Append() calls
I20260928 06:31:03.846967 3234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb7bf101bf764f27960d5a5ad1825a73: No bootstrap required, opened a new log
I20260928 06:31:03.849136 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 3166
I20260928 06:31:03.849241 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/master-0/wal/instance
I20260928 06:31:03.849426 23430 external_mini_cluster.cc:1428] Setting key 1b9a1b216e03cd09a87748ca58644d8c
I20260928 06:31:03.849295 3234 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb7bf101bf764f27960d5a5ad1825a73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb7bf101bf764f27960d5a5ad1825a73" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 43353 } }
I20260928 06:31:03.849483 3234 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb7bf101bf764f27960d5a5ad1825a73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:03.849593 3234 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb7bf101bf764f27960d5a5ad1825a73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb7bf101bf764f27960d5a5ad1825a73, State: Initialized, Role: FOLLOWER
I20260928 06:31:03.849761 3234 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb7bf101bf764f27960d5a5ad1825a73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb7bf101bf764f27960d5a5ad1825a73" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 43353 } }
I20260928 06:31:03.849861 3234 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb7bf101bf764f27960d5a5ad1825a73 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:31:03.849933 3234 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb7bf101bf764f27960d5a5ad1825a73 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:31:03.849994 3234 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb7bf101bf764f27960d5a5ad1825a73 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:03.850788 3234 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb7bf101bf764f27960d5a5ad1825a73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb7bf101bf764f27960d5a5ad1825a73" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 43353 } }
I20260928 06:31:03.850906 3234 leader_election.cc:302] T 00000000000000000000000000000000 P fb7bf101bf764f27960d5a5ad1825a73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb7bf101bf764f27960d5a5ad1825a73; no voters:
I20260928 06:31:03.851126 3234 leader_election.cc:288] T 00000000000000000000000000000000 P fb7bf101bf764f27960d5a5ad1825a73 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:31:03.851156 3239 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb7bf101bf764f27960d5a5ad1825a73 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:31:03.851392 3239 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb7bf101bf764f27960d5a5ad1825a73 [term 1 LEADER]: Becoming Leader. State: Replica: fb7bf101bf764f27960d5a5ad1825a73, State: Running, Role: LEADER
I20260928 06:31:03.851492 3234 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb7bf101bf764f27960d5a5ad1825a73 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:31:03.851518 3239 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb7bf101bf764f27960d5a5ad1825a73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb7bf101bf764f27960d5a5ad1825a73" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 43353 } }
I20260928 06:31:03.851854 3240 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb7bf101bf764f27960d5a5ad1825a73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb7bf101bf764f27960d5a5ad1825a73. Latest consensus state: current_term: 1 leader_uuid: "fb7bf101bf764f27960d5a5ad1825a73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb7bf101bf764f27960d5a5ad1825a73" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 43353 } } }
I20260928 06:31:03.852072 3240 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb7bf101bf764f27960d5a5ad1825a73 [sys.catalog]: This master's current role is: LEADER
I20260928 06:31:03.851985 3241 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb7bf101bf764f27960d5a5ad1825a73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb7bf101bf764f27960d5a5ad1825a73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb7bf101bf764f27960d5a5ad1825a73" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 43353 } } }
I20260928 06:31:03.852308 3241 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb7bf101bf764f27960d5a5ad1825a73 [sys.catalog]: This master's current role is: LEADER
I20260928 06:31:03.852479 3243 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:31:03.853076 3243 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:31:03.853416 3234 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 06:31:03.858798 3243 catalog_manager.cc:1347] Generated new cluster ID: f6cead3f1e0d47589f6ad5619f67cac5
I20260928 06:31:03.858930 3243 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:31:03 dist-test-slave-d59m krb5kdc[2722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:31:03.865962 3238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:03.850102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49586 (local address 127.22.225.190:43353)
0928 06:31:03.850349 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:03.850359 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:03.850395 (+ 36us) server_negotiation.cc:407] Connection header received
0928 06:31:03.850512 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:03.850521 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:03.850582 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:03.850728 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:03.851565 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:03.852987 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:03.855274 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:03.857007 (+ 1733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:03.857162 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:03.860452 (+ 3290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:03.860457 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:03.860458 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:03.860471 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:03.863146 (+ 2675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:03.863589 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:03.863592 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:03.863592 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:03.863733 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:03.863972 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:03.863973 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:03.863973 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:03.864327 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:03.864478 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:03.865099 (+ 621us) server_negotiation.cc:299] Negotiation successful
0928 06:31:03.865304 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":89,"threads_started":1}
I20260928 06:31:03.869944 3243 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:31:03.870733 3243 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:31:03.876399 3243 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb7bf101bf764f27960d5a5ad1825a73: Generated new TSK 0
I20260928 06:31:03.876638 3243 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.225.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab.
Entry for principal kudu/127.22.225.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab.
I20260928 06:31:04.960984 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.225.148:51731/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.129:0
--local_ip_for_outbound_sockets=127.22.225.129
--webserver_interface=127.22.225.129
--webserver_port=0
--tserver_master_addrs=127.22.225.190:43353
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab
--principal=kudu/127.22.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:44983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:31:05.090636 3279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:05.091009 3279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:05.091099 3279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:05.093102 3279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:05.093218 3279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:05.093307 3279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.129
I20260928 06:31:05.095860 3279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:44983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.22.225.148:51731/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab
--principal=kudu/127.22.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.225.129
--webserver_port=0
--tserver_master_addrs=127.22.225.190:43353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.129
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:31:05.096190 3279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:05.096563 3279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:05.103788 3284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:05.103909 3285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:05.103808 3287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:05.105139 3279 server_base.cc:1034] running on GCE node
I20260928 06:31:05.105404 3279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:05.105734 3279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:05.113842 3279 hybrid_clock.cc:648] HybridClock initialized: now 1790577065113800 us; error 52 us; skew 500 ppm
Sep 28 06:31:05 dist-test-slave-d59m krb5kdc[2722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577065, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:05.117527 3279 init.cc:377] Logged in from keytab as kudu/127.22.225.129@KRBTEST.COM (short username kudu)
I20260928 06:31:05.118151 3279 webserver.cc:460] Webserver started at http://127.22.225.129:34965/ using document root <none> and password file <none>
I20260928 06:31:05.118486 3279 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:05.118563 3279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:05.118745 3279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 06:31:05 dist-test-slave-d59m krb5kdc[2722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577065, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.129@KRBTEST.COM for HTTP/127.22.225.148@KRBTEST.COM
I20260928 06:31:05.158555 3279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-0/data/instance:
uuid: "ccc451df5c534f498bd5fafd19553547"
format_stamp: "Formatted at 2026-09-28 06:31:05 on dist-test-slave-d59m"
server_key: "c2c840ee2aae497eb8dafb437543692c"
server_key_iv: "12038d92421dfd55e1ee97753243dd9e"
server_key_version: "kuduclusterkey@0"
I20260928 06:31:05.158977 3279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-0/wal/instance:
uuid: "ccc451df5c534f498bd5fafd19553547"
format_stamp: "Formatted at 2026-09-28 06:31:05 on dist-test-slave-d59m"
server_key: "c2c840ee2aae497eb8dafb437543692c"
server_key_iv: "12038d92421dfd55e1ee97753243dd9e"
server_key_version: "kuduclusterkey@0"
I20260928 06:31:05.160941 3279 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.001s
I20260928 06:31:05.189064 3296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:05.189366 3279 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260928 06:31:05.189466 3279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-0/wal
uuid: "ccc451df5c534f498bd5fafd19553547"
format_stamp: "Formatted at 2026-09-28 06:31:05 on dist-test-slave-d59m"
server_key: "c2c840ee2aae497eb8dafb437543692c"
server_key_iv: "12038d92421dfd55e1ee97753243dd9e"
server_key_version: "kuduclusterkey@0"
I20260928 06:31:05.189580 3279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:05.223444 3279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:05.233393 3279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:05.233621 3279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:05.233911 3279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:31:05.234344 3279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:31:05.234429 3279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:05.234524 3279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:31:05.234591 3279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:05.245611 3279 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.129:43865
I20260928 06:31:05.246264 3279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
I20260928 06:31:05.251187 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 3279
I20260928 06:31:05.251293 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-0/wal/instance
I20260928 06:31:05.251472 23430 external_mini_cluster.cc:1428] Setting key e8e26ac40084635492f0d1695f694306
I20260928 06:31:05.249001 3409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.129:43865 every 8 connection(s)
Sep 28 06:31:05 dist-test-slave-d59m krb5kdc[2722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577065, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.129@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:31:05.264932 3412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:05.247420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:43353 (local address 127.22.225.129:46213)
0928 06:31:05.249345 (+ 1925us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:05.249396 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:05.250205 (+ 809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:05.250631 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:05.250635 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:05.250909 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:05.251308 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:05.251317 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:05.252528 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:05.252529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:05.254166 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:05.254170 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:05.254792 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:05.254794 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:05.254947 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:05.255590 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:05.255601 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:05.257970 (+ 2369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:05.263185 (+ 5215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:05.263195 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:05.263198 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:05.263530 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:05.263749 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:05.263750 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:05.263750 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:05.263878 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:05.264192 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:05.264201 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:05.264385 (+ 184us) client_negotiation.cc:769] Sending connection context
0928 06:31:05.264502 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 06:31:05.264688 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1874,"thread_start_us":99,"threads_started":1}
I20260928 06:31:05.265316 3413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:05.247435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:46213 (local address 127.22.225.190:43353)
0928 06:31:05.249267 (+ 1832us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:05.249270 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:05.250367 (+ 1097us) server_negotiation.cc:407] Connection header received
0928 06:31:05.250404 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:05.250405 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:05.250421 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:05.250487 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:05.251427 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:05.252426 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:05.254277 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:05.254636 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:05.254704 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:05.258105 (+ 3401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:05.258107 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:05.258107 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:05.258116 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:05.263065 (+ 4949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:05.263614 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:05.263615 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:05.263615 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:05.263684 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:05.263942 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:05.263942 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:05.263943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:05.264086 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:05.264161 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:05.265105 (+ 944us) server_negotiation.cc:299] Negotiation successful
0928 06:31:05.265213 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1796,"thread_start_us":107,"threads_started":1}
I20260928 06:31:05.265904 3410 heartbeater.cc:347] Connected to a master server at 127.22.225.190:43353
I20260928 06:31:05.266044 3410 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:05.266284 3410 heartbeater.cc:511] Master 127.22.225.190:43353 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.22.225.130@KRBTEST.COM; defaulting to no policy
I20260928 06:31:05.267241 3198 ts_manager.cc:194] Registered new tserver with Master: ccc451df5c534f498bd5fafd19553547 (127.22.225.129:43865)
I20260928 06:31:05.268500 3198 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.225.129@KRBTEST.COM'} at 127.22.225.129:46213
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab.
Entry for principal kudu/127.22.225.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab.
I20260928 06:31:05.312775 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.225.148:51731/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.130:0
--local_ip_for_outbound_sockets=127.22.225.130
--webserver_interface=127.22.225.130
--webserver_port=0
--tserver_master_addrs=127.22.225.190:43353
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab
--principal=kudu/127.22.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:44983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 06:31:05 dist-test-slave-d59m krb5kdc[2722](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790577065, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 06:31:05.442131 3418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:05.442430 3418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:05.442472 3418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:05.444756 3418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:05.444868 3418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:05.444950 3418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.130
I20260928 06:31:05.447314 3418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:44983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.22.225.148:51731/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab
--principal=kudu/127.22.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.225.130
--webserver_port=0
--tserver_master_addrs=127.22.225.190:43353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.130
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:31:05.447588 3418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:05.447918 3418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:05.455127 3424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:05.455549 3423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:05.455724 3426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:05.456923 3418 server_base.cc:1034] running on GCE node
I20260928 06:31:05.457175 3418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:05.457454 3418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:05.458644 3418 hybrid_clock.cc:648] HybridClock initialized: now 1790577065458622 us; error 33 us; skew 500 ppm
Sep 28 06:31:05 dist-test-slave-d59m krb5kdc[2722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577065, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:05.461283 3418 init.cc:377] Logged in from keytab as kudu/127.22.225.130@KRBTEST.COM (short username kudu)
I20260928 06:31:05.461854 3418 webserver.cc:460] Webserver started at http://127.22.225.130:34361/ using document root <none> and password file <none>
I20260928 06:31:05.462085 3418 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:05.462153 3418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:05.462308 3418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 06:31:05 dist-test-slave-d59m krb5kdc[2722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577065, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.130@KRBTEST.COM for HTTP/127.22.225.148@KRBTEST.COM
I20260928 06:31:05.480908 3418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-1/data/instance:
uuid: "a19efcfb962f45b58fc10d8d60a1fea6"
format_stamp: "Formatted at 2026-09-28 06:31:05 on dist-test-slave-d59m"
server_key: "c0a60323433808a5f3af4b092de267d2"
server_key_iv: "86f15e6bd25a7e3858440785029d08a3"
server_key_version: "kuduclusterkey@0"
I20260928 06:31:05.481475 3418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-1/wal/instance:
uuid: "a19efcfb962f45b58fc10d8d60a1fea6"
format_stamp: "Formatted at 2026-09-28 06:31:05 on dist-test-slave-d59m"
server_key: "c0a60323433808a5f3af4b092de267d2"
server_key_iv: "86f15e6bd25a7e3858440785029d08a3"
server_key_version: "kuduclusterkey@0"
I20260928 06:31:05.498786 3418 fs_manager.cc:691] Time spent creating directory manager: real 0.017s user 0.002s sys 0.000s
I20260928 06:31:05.536741 3434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:05.537060 3418 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.000s sys 0.000s
I20260928 06:31:05.537284 3418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-1/wal
uuid: "a19efcfb962f45b58fc10d8d60a1fea6"
format_stamp: "Formatted at 2026-09-28 06:31:05 on dist-test-slave-d59m"
server_key: "c0a60323433808a5f3af4b092de267d2"
server_key_iv: "86f15e6bd25a7e3858440785029d08a3"
server_key_version: "kuduclusterkey@0"
I20260928 06:31:05.537478 3418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:05.558056 3418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:05.565987 3418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:05.566247 3418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:05.566571 3418 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:31:05.567049 3418 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:31:05.567159 3418 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:05.567250 3418 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:31:05.567315 3418 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:05.580562 3418 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.130:43203
I20260928 06:31:05.581282 3418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
I20260928 06:31:05.582741 3547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.130:43203 every 8 connection(s)
I20260928 06:31:05.583395 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 3418
I20260928 06:31:05.583521 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-1/wal/instance
I20260928 06:31:05.583755 23430 external_mini_cluster.cc:1428] Setting key ea8c29096912228fd985612307c84df8
WARNING: no policy specified for kudu/127.22.225.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.131@KRBTEST.COM" created.
Sep 28 06:31:05 dist-test-slave-d59m krb5kdc[2722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577065, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.130@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:31:05.624907 3413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:05.594464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:38185 (local address 127.22.225.190:43353)
0928 06:31:05.594532 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:05.594534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:05.603327 (+ 8793us) server_negotiation.cc:407] Connection header received
0928 06:31:05.604230 (+ 903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:05.604231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:05.604245 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:05.604293 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:05.614125 (+ 9832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:05.615086 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:05.616890 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:05.617176 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:05.617240 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:05.621185 (+ 3945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:05.621187 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:05.621188 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:05.621197 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:05.623255 (+ 2058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:05.623761 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:05.623763 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:05.623763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:05.623823 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:05.624104 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:05.624104 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:05.624105 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:05.624257 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:05.624309 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:05.624714 (+ 405us) server_negotiation.cc:299] Negotiation successful
0928 06:31:05.624810 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 06:31:05.625531 3551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:05.594093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:43353 (local address 127.22.225.130:38185)
0928 06:31:05.603113 (+ 9020us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:05.603199 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:05.604103 (+ 904us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:05.604383 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:05.604388 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:05.604702 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:05.613951 (+ 9249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:05.613966 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:05.615248 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:05.615250 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:05.616771 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:05.616775 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:05.617359 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:05.617359 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:05.617502 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:05.619367 (+ 1865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:05.619378 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:05.621059 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:05.623357 (+ 2298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:05.623365 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:05.623368 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:05.623667 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:05.623891 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:05.623892 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:05.623892 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:05.624030 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:05.624346 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:05.624353 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:05.624528 (+ 175us) client_negotiation.cc:769] Sending connection context
0928 06:31:05.625074 (+ 546us) client_negotiation.cc:240] Negotiation successful
0928 06:31:05.625268 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8940,"thread_start_us":120,"threads_started":1}
I20260928 06:31:05.627110 3548 heartbeater.cc:347] Connected to a master server at 127.22.225.190:43353
I20260928 06:31:05.627236 3548 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:05.627453 3548 heartbeater.cc:511] Master 127.22.225.190:43353 requested a full tablet report, sending...
I20260928 06:31:05.627876 3198 ts_manager.cc:194] Registered new tserver with Master: a19efcfb962f45b58fc10d8d60a1fea6 (127.22.225.130:43203)
I20260928 06:31:05.628423 3198 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.225.130@KRBTEST.COM'} at 127.22.225.130:38185
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab.
Entry for principal kudu/127.22.225.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab.
I20260928 06:31:05.660674 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.225.148:51731/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.131:0
--local_ip_for_outbound_sockets=127.22.225.131
--webserver_interface=127.22.225.131
--webserver_port=0
--tserver_master_addrs=127.22.225.190:43353
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab
--principal=kudu/127.22.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:44983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:31:05.834059 3553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:05.834316 3553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:05.834355 3553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:05.835985 3553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:05.836040 3553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:05.836079 3553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.131
I20260928 06:31:05.838213 3553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:44983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.22.225.148:51731/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab
--principal=kudu/127.22.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.225.131
--webserver_port=0
--tserver_master_addrs=127.22.225.190:43353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.131
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:31:05.838451 3553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:05.838717 3553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:05.863008 3561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:05.863009 3559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:05.864276 3553 server_base.cc:1034] running on GCE node
W20260928 06:31:05.864442 3558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:05.864723 3553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:05.865027 3553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:05.896637 3553 hybrid_clock.cc:648] HybridClock initialized: now 1790577065896608 us; error 78 us; skew 500 ppm
Sep 28 06:31:05 dist-test-slave-d59m krb5kdc[2722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577065, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:05.912045 3553 init.cc:377] Logged in from keytab as kudu/127.22.225.131@KRBTEST.COM (short username kudu)
I20260928 06:31:05.912717 3553 webserver.cc:460] Webserver started at http://127.22.225.131:45075/ using document root <none> and password file <none>
I20260928 06:31:05.913209 3553 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:05.913282 3553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:05.913424 3553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 06:31:05 dist-test-slave-d59m krb5kdc[2722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577065, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.131@KRBTEST.COM for HTTP/127.22.225.148@KRBTEST.COM
I20260928 06:31:05.940308 3553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-2/data/instance:
uuid: "73660479c22e4ebe977f06eb4cffab76"
format_stamp: "Formatted at 2026-09-28 06:31:05 on dist-test-slave-d59m"
server_key: "b92a6eeb9740bf0ae8e8fe144d6bde1e"
server_key_iv: "7c3c72fb608e968b377898100159bdd7"
server_key_version: "kuduclusterkey@0"
I20260928 06:31:05.940871 3553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-2/wal/instance:
uuid: "73660479c22e4ebe977f06eb4cffab76"
format_stamp: "Formatted at 2026-09-28 06:31:05 on dist-test-slave-d59m"
server_key: "b92a6eeb9740bf0ae8e8fe144d6bde1e"
server_key_iv: "7c3c72fb608e968b377898100159bdd7"
server_key_version: "kuduclusterkey@0"
I20260928 06:31:05.947223 3553 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.001s
I20260928 06:31:05.972236 3568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:05.972538 3553 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.000s sys 0.000s
I20260928 06:31:05.972666 3553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-2/wal
uuid: "73660479c22e4ebe977f06eb4cffab76"
format_stamp: "Formatted at 2026-09-28 06:31:05 on dist-test-slave-d59m"
server_key: "b92a6eeb9740bf0ae8e8fe144d6bde1e"
server_key_iv: "7c3c72fb608e968b377898100159bdd7"
server_key_version: "kuduclusterkey@0"
I20260928 06:31:05.972775 3553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:05.997496 3553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:06.014297 3553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:06.014453 3553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:06.014679 3553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:31:06.015049 3553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:31:06.015084 3553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:06.015115 3553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:31:06.015131 3553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:06.022725 3553 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.131:36903
I20260928 06:31:06.027045 3553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
I20260928 06:31:06.032820 3681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.131:36903 every 8 connection(s)
I20260928 06:31:06.042750 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 3553
I20260928 06:31:06.042903 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-2/wal/instance
I20260928 06:31:06.043129 23430 external_mini_cluster.cc:1428] Setting key 930044c1bd6a9520c2c2d43e6741f434
Sep 28 06:31:06 dist-test-slave-d59m krb5kdc[2722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577065, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.131@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:31:06.079838 3684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:06.037521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:43353 (local address 127.22.225.131:38591)
0928 06:31:06.043499 (+ 5978us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:06.043571 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:06.044295 (+ 724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:06.044704 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:06.044709 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:06.044994 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:06.045296 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:06.045302 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:06.046369 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:06.046370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:06.047665 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:06.047668 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:06.048250 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:06.048251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:06.048398 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:06.056756 (+ 8358us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:06.056770 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:06.058521 (+ 1751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:06.078084 (+ 19563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:06.078107 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:06.078110 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:06.078469 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:06.078692 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:06.078693 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:06.078694 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:06.078842 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:06.079152 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:06.079159 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:06.079334 (+ 175us) client_negotiation.cc:769] Sending connection context
0928 06:31:06.079431 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 06:31:06.079593 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5908,"thread_start_us":113,"threads_started":1}
I20260928 06:31:06.080602 3413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:06.037865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:38591 (local address 127.22.225.190:43353)
0928 06:31:06.037917 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:06.037919 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:06.043679 (+ 5760us) server_negotiation.cc:407] Connection header received
0928 06:31:06.044430 (+ 751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:06.044431 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:06.044443 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:06.044474 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:06.045409 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:06.046240 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:06.047775 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:06.048079 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:06.048135 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:06.058662 (+ 10527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:06.058663 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:06.058664 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:06.058673 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:06.077966 (+ 19293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:06.078558 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:06.078559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:06.078559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:06.078626 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:06.078906 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:06.078906 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:06.078907 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:06.079059 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:06.079125 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:06.080394 (+ 1269us) server_negotiation.cc:299] Negotiation successful
0928 06:31:06.080488 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 06:31:06.081084 3682 heartbeater.cc:347] Connected to a master server at 127.22.225.190:43353
I20260928 06:31:06.081213 3682 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:06.081429 3682 heartbeater.cc:511] Master 127.22.225.190:43353 requested a full tablet report, sending...
I20260928 06:31:06.081853 3198 ts_manager.cc:194] Registered new tserver with Master: 73660479c22e4ebe977f06eb4cffab76 (127.22.225.131:36903)
I20260928 06:31:06.082351 3198 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.225.131@KRBTEST.COM'} at 127.22.225.131:38591
I20260928 06:31:06.087458 23430 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:31:06.088358 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 3279
I20260928 06:31:06.095642 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 3418
I20260928 06:31:06.102713 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 3553
I20260928 06:31:06.110016 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 3166
I20260928 06:31:06.116235 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:43353
--webserver_interface=127.22.225.190
--webserver_port=40811
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab
--principal=kudu/127.22.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:44983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.225.190:43353
--ranger_config_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.225.148:51731/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.22.225.148:51731/kms with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:31:06.257165 3685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:06.257529 3685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:06.257613 3685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:06.259464 3685 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:31:06.259573 3685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:06.259651 3685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:06.259713 3685 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:31:06.259773 3685 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:31:06.262296 3685 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:44983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.22.225.148:51731/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.22.225.190:43353
--ranger_config_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab
--principal=kudu/127.22.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:43353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.225.190
--webserver_port=40811
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:31:06.262818 3685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:06.263235 3685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:06.269145 3691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:06.269160 3690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:06.272809 3685 server_base.cc:1034] running on GCE node
W20260928 06:31:06.273216 3693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:06.273562 3685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:06.274017 3685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:06.275225 3685 hybrid_clock.cc:648] HybridClock initialized: now 1790577066275212 us; error 36 us; skew 500 ppm
Sep 28 06:31:06 dist-test-slave-d59m krb5kdc[2722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577066, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:06.279400 3685 init.cc:377] Logged in from keytab as kudu/127.22.225.190@KRBTEST.COM (short username kudu)
I20260928 06:31:06.280063 3685 webserver.cc:460] Webserver started at http://127.22.225.190:40811/ using document root <none> and password file <none>
I20260928 06:31:06.280387 3685 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:06.280501 3685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 06:31:06 dist-test-slave-d59m krb5kdc[2722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577066, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.190@KRBTEST.COM for HTTP/127.22.225.148@KRBTEST.COM
I20260928 06:31:06.347470 3685 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:06.348724 3701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:06.349025 3685 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:06.349203 3685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/master-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/master-0/wal
uuid: "fb7bf101bf764f27960d5a5ad1825a73"
format_stamp: "Formatted at 2026-09-28 06:31:03 on dist-test-slave-d59m"
server_key: "31b0310b4429e723825d62e0724e67a6"
server_key_iv: "84a1aeb7293508815b744c8019719843"
server_key_version: "kuduclusterkey@0"
I20260928 06:31:06.349884 3685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:06.377007 3685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:06.379931 3685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:06.380208 3685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:06.387741 3685 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.190:43353
I20260928 06:31:06.387787 3753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.190:43353 every 8 connection(s)
I20260928 06:31:06.388478 3685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
I20260928 06:31:06.389783 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 3685
I20260928 06:31:06.390468 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.225.148:51731/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.129:43865
--local_ip_for_outbound_sockets=127.22.225.129
--tserver_master_addrs=127.22.225.190:43353
--webserver_port=34965
--webserver_interface=127.22.225.129
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab
--principal=kudu/127.22.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:44983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.22.225.148:51731/kms with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:31:06.394780 3755 sys_catalog.cc:263] Verifying existing consensus state
I20260928 06:31:06.395978 3755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb7bf101bf764f27960d5a5ad1825a73: Bootstrap starting.
I20260928 06:31:06.413689 3755 log.cc:824] T 00000000000000000000000000000000 P fb7bf101bf764f27960d5a5ad1825a73: Log is configured to *not* fsync() on all Append() calls
I20260928 06:31:06.416982 3755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb7bf101bf764f27960d5a5ad1825a73: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:31:06.417325 3755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb7bf101bf764f27960d5a5ad1825a73: Bootstrap complete.
I20260928 06:31:06.421463 3755 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb7bf101bf764f27960d5a5ad1825a73 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb7bf101bf764f27960d5a5ad1825a73" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 43353 } }
I20260928 06:31:06.421829 3755 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb7bf101bf764f27960d5a5ad1825a73 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb7bf101bf764f27960d5a5ad1825a73, State: Initialized, Role: FOLLOWER
I20260928 06:31:06.422019 3755 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb7bf101bf764f27960d5a5ad1825a73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb7bf101bf764f27960d5a5ad1825a73" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 43353 } }
I20260928 06:31:06.422096 3755 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb7bf101bf764f27960d5a5ad1825a73 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:31:06.422144 3755 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb7bf101bf764f27960d5a5ad1825a73 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:31:06.422188 3755 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb7bf101bf764f27960d5a5ad1825a73 [term 1 FOLLOWER]: Advancing to term 2
I20260928 06:31:06.428792 3755 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb7bf101bf764f27960d5a5ad1825a73 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb7bf101bf764f27960d5a5ad1825a73" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 43353 } }
I20260928 06:31:06.428962 3755 leader_election.cc:302] T 00000000000000000000000000000000 P fb7bf101bf764f27960d5a5ad1825a73 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb7bf101bf764f27960d5a5ad1825a73; no voters:
I20260928 06:31:06.429172 3755 leader_election.cc:288] T 00000000000000000000000000000000 P fb7bf101bf764f27960d5a5ad1825a73 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 06:31:06.429251 3758 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb7bf101bf764f27960d5a5ad1825a73 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 06:31:06.429443 3758 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb7bf101bf764f27960d5a5ad1825a73 [term 2 LEADER]: Becoming Leader. State: Replica: fb7bf101bf764f27960d5a5ad1825a73, State: Running, Role: LEADER
I20260928 06:31:06.429491 3755 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb7bf101bf764f27960d5a5ad1825a73 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:31:06.429563 3758 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb7bf101bf764f27960d5a5ad1825a73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb7bf101bf764f27960d5a5ad1825a73" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 43353 } }
I20260928 06:31:06.429873 3759 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb7bf101bf764f27960d5a5ad1825a73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "fb7bf101bf764f27960d5a5ad1825a73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb7bf101bf764f27960d5a5ad1825a73" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 43353 } } }
I20260928 06:31:06.429872 3758 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb7bf101bf764f27960d5a5ad1825a73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb7bf101bf764f27960d5a5ad1825a73. Latest consensus state: current_term: 2 leader_uuid: "fb7bf101bf764f27960d5a5ad1825a73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb7bf101bf764f27960d5a5ad1825a73" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 43353 } } }
I20260928 06:31:06.429983 3759 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb7bf101bf764f27960d5a5ad1825a73 [sys.catalog]: This master's current role is: LEADER
I20260928 06:31:06.429988 3758 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb7bf101bf764f27960d5a5ad1825a73 [sys.catalog]: This master's current role is: LEADER
I20260928 06:31:06.430186 3764 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:31:06.430735 3764 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:31:06.430895 3764 catalog_manager.cc:1259] Loaded cluster ID: f6cead3f1e0d47589f6ad5619f67cac5
I20260928 06:31:06.430938 3764 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:31:06.432233 3764 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:31:06.432442 3764 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fb7bf101bf764f27960d5a5ad1825a73: Loaded TSK: 0
I20260928 06:31:06.432698 3764 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:31:06.520856 3755 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260928 06:31:06.542181 3754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:06.542584 3754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:06.542670 3754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:06.544641 3754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:06.544757 3754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:06.544840 3754 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.129
I20260928 06:31:06.547106 3754 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:44983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.22.225.148:51731/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab
--principal=kudu/127.22.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.129:43865
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.225.129
--webserver_port=34965
--tserver_master_addrs=127.22.225.190:43353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3754
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.129
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:31:06.547701 3754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:06.548050 3754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:06.556421 3786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:06.556502 3784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:06.557232 3783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:06.558099 3754 server_base.cc:1034] running on GCE node
I20260928 06:31:06.558470 3754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:06.558784 3754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:06.562057 3754 hybrid_clock.cc:648] HybridClock initialized: now 1790577066562015 us; error 53 us; skew 500 ppm
Sep 28 06:31:06 dist-test-slave-d59m krb5kdc[2722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577066, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:06.566133 3754 init.cc:377] Logged in from keytab as kudu/127.22.225.129@KRBTEST.COM (short username kudu)
I20260928 06:31:06.566817 3754 webserver.cc:460] Webserver started at http://127.22.225.129:34965/ using document root <none> and password file <none>
I20260928 06:31:06.567148 3754 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:06.567284 3754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 06:31:06 dist-test-slave-d59m krb5kdc[2722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577066, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.129@KRBTEST.COM for HTTP/127.22.225.148@KRBTEST.COM
I20260928 06:31:06.613993 3754 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:31:06.623136 3795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:06.630290 3754 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.000s sys 0.001s
I20260928 06:31:06.630443 3754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-0/wal
uuid: "ccc451df5c534f498bd5fafd19553547"
format_stamp: "Formatted at 2026-09-28 06:31:05 on dist-test-slave-d59m"
server_key: "c2c840ee2aae497eb8dafb437543692c"
server_key_iv: "12038d92421dfd55e1ee97753243dd9e"
server_key_version: "kuduclusterkey@0"
I20260928 06:31:06.631012 3754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:06.666426 3754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:06.669181 3754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:06.669339 3754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:06.669605 3754 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:31:06.669963 3754 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:31:06.670009 3754 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:06.670046 3754 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:31:06.670069 3754 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:06.683773 3754 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.129:43865
I20260928 06:31:06.683822 3910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.129:43865 every 8 connection(s)
I20260928 06:31:06.684280 3754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
I20260928 06:31:06.693848 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 3754
Sep 28 06:31:06 dist-test-slave-d59m krb5kdc[2722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577066, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.129@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:31:06.703009 3913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:06.685076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:43353 (local address 127.22.225.129:57741)
0928 06:31:06.685246 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:06.685297 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:06.686273 (+ 976us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:06.687019 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:06.687024 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:06.687268 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:06.687572 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:06.687579 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:06.689425 (+ 1846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:06.689426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:06.690690 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:06.690693 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:06.691626 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:06.691627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:06.691769 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:06.696781 (+ 5012us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:06.696796 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:06.698471 (+ 1675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:06.700971 (+ 2500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:06.700983 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:06.700985 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:06.701309 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:06.701660 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:06.701661 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:06.701661 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:06.701796 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:06.702288 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:06.702296 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:06.702455 (+ 159us) client_negotiation.cc:769] Sending connection context
0928 06:31:06.702565 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 06:31:06.702748 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":87,"threads_started":1}
I20260928 06:31:06.703658 3914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:06.685510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:57741 (local address 127.22.225.190:43353)
0928 06:31:06.686503 (+ 993us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:06.686513 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:06.686562 (+ 49us) server_negotiation.cc:407] Connection header received
0928 06:31:06.686726 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:06.686734 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:06.686795 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:06.687717 (+ 922us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:06.687830 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:06.689296 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:06.690812 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:06.691431 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:06.692075 (+ 644us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:06.698675 (+ 6600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:06.698682 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:06.698682 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:06.698697 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:06.700827 (+ 2130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:06.701425 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:06.701428 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:06.701428 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:06.701568 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:06.701867 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:06.701868 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:06.701868 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:06.702192 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:06.703207 (+ 1015us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:06.703288 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 06:31:06.703420 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":925,"thread_start_us":102,"threads_started":1}
I20260928 06:31:06.704381 3911 heartbeater.cc:347] Connected to a master server at 127.22.225.190:43353
I20260928 06:31:06.704497 3911 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:06.704732 3911 heartbeater.cc:511] Master 127.22.225.190:43353 requested a full tablet report, sending...
W20260928 06:31:06.705670 3911 heartbeater.cc:644] Failed to heartbeat to 127.22.225.190:43353 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 06:31:06.705746 3911 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:06.705806 3911 heartbeater.cc:511] Master 127.22.225.190:43353 requested a full tablet report, sending...
I20260928 06:31:06.706133 3911 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:06.706218 3911 heartbeater.cc:511] Master 127.22.225.190:43353 requested a full tablet report, sending...
W20260928 06:31:06.706534 3911 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 06:31:06.708091 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.225.148:51731/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.130:43203
--local_ip_for_outbound_sockets=127.22.225.130
--tserver_master_addrs=127.22.225.190:43353
--webserver_port=34361
--webserver_interface=127.22.225.130
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab
--principal=kudu/127.22.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:44983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.22.225.148:51731/kms with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:31:06.850236 3916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:06.850608 3916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:06.850709 3916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:06.852658 3916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:06.852783 3916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:06.852867 3916 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.130
I20260928 06:31:06.855441 3916 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:44983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.22.225.148:51731/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab
--principal=kudu/127.22.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.130:43203
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.225.130
--webserver_port=34361
--tserver_master_addrs=127.22.225.190:43353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3916
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.130
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:31:06.855967 3916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:06.856336 3916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:06.862823 3926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:06.862793 3928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:06.863302 3925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:06.864172 3916 server_base.cc:1034] running on GCE node
I20260928 06:31:06.864410 3916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:06.864734 3916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:06.865974 3916 hybrid_clock.cc:648] HybridClock initialized: now 1790577066865954 us; error 37 us; skew 500 ppm
Sep 28 06:31:06 dist-test-slave-d59m krb5kdc[2722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577066, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:06.879317 3916 init.cc:377] Logged in from keytab as kudu/127.22.225.130@KRBTEST.COM (short username kudu)
I20260928 06:31:06.879951 3916 webserver.cc:460] Webserver started at http://127.22.225.130:34361/ using document root <none> and password file <none>
I20260928 06:31:06.880240 3916 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:06.880326 3916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 06:31:06 dist-test-slave-d59m krb5kdc[2722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577066, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.130@KRBTEST.COM for HTTP/127.22.225.148@KRBTEST.COM
I20260928 06:31:06.918855 3916 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:06.920002 3936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:06.920215 3916 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:06.920326 3916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-1/wal
uuid: "a19efcfb962f45b58fc10d8d60a1fea6"
format_stamp: "Formatted at 2026-09-28 06:31:05 on dist-test-slave-d59m"
server_key: "c0a60323433808a5f3af4b092de267d2"
server_key_iv: "86f15e6bd25a7e3858440785029d08a3"
server_key_version: "kuduclusterkey@0"
I20260928 06:31:06.920850 3916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:06.936928 3916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:06.939810 3916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:06.940056 3916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:06.940569 3916 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:31:06.941107 3916 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:31:06.941179 3916 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:06.941220 3916 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:31:06.941246 3916 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:06.966625 3916 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.130:43203
I20260928 06:31:06.966715 4049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.130:43203 every 8 connection(s)
I20260928 06:31:06.967207 3916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
Sep 28 06:31:06 dist-test-slave-d59m krb5kdc[2722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577066, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.130@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:31:06.977578 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 3916
I20260928 06:31:06.977911 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.225.148:51731/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.131:36903
--local_ip_for_outbound_sockets=127.22.225.131
--tserver_master_addrs=127.22.225.190:43353
--webserver_port=45075
--webserver_interface=127.22.225.131
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab
--principal=kudu/127.22.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:44983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.22.225.148:51731/kms with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:31:06.996143 3914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:06.968323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:51043 (local address 127.22.225.190:43353)
0928 06:31:06.968370 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:06.968372 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:06.968384 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:31:06.969033 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:06.969034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:06.969051 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:06.969105 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:06.969866 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:06.970677 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:06.972164 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:06.972474 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:06.972555 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:06.977776 (+ 5221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:06.977778 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:06.977778 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:06.977788 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:06.994361 (+ 16573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:06.994977 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:06.994978 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:06.994979 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:06.995051 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:06.995363 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:06.995364 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:06.995364 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:06.995525 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:06.995600 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:06.995932 (+ 332us) server_negotiation.cc:299] Negotiation successful
0928 06:31:06.996044 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 06:31:06.996352 4052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:06.968059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:43353 (local address 127.22.225.130:51043)
0928 06:31:06.968220 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:06.968264 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:06.968915 (+ 651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:06.969180 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:06.969184 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:06.969480 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:06.969769 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:06.969775 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:06.970773 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:06.970775 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:06.972079 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:06.972083 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:06.972600 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:06.972601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:06.972745 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:06.973379 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:06.973389 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:06.977669 (+ 4280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:06.994493 (+ 16824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:06.994509 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:06.994512 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:06.994886 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:06.995115 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:06.995116 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:06.995116 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:06.995303 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:06.995589 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:06.995598 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:06.995790 (+ 192us) client_negotiation.cc:769] Sending connection context
0928 06:31:06.995913 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 06:31:06.996088 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":83,"threads_started":1}
I20260928 06:31:06.996848 4050 heartbeater.cc:347] Connected to a master server at 127.22.225.190:43353
I20260928 06:31:06.996953 4050 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:06.997148 4050 heartbeater.cc:511] Master 127.22.225.190:43353 requested a full tablet report, sending...
W20260928 06:31:06.997777 4050 heartbeater.cc:644] Failed to heartbeat to 127.22.225.190:43353 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 06:31:06.997843 4050 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:06.997903 4050 heartbeater.cc:511] Master 127.22.225.190:43353 requested a full tablet report, sending...
I20260928 06:31:06.998224 4050 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:06.998292 4050 heartbeater.cc:511] Master 127.22.225.190:43353 requested a full tablet report, sending...
W20260928 06:31:06.998575 4050 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 06:31:07.142988 4053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:07.143385 4053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:07.143486 4053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:07.145578 4053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:07.145795 4053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:07.145892 4053 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.131
I20260928 06:31:07.148083 4053 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:44983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.22.225.148:51731/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab
--principal=kudu/127.22.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.131:36903
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.225.131
--webserver_port=45075
--tserver_master_addrs=127.22.225.190:43353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4053
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.131
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:31:07.148684 4053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:07.149044 4053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:07.155568 4059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:07.155982 4058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:07.156211 4053 server_base.cc:1034] running on GCE node
W20260928 06:31:07.156937 4061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:07.157297 4053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:07.157610 4053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:07.158756 4053 hybrid_clock.cc:648] HybridClock initialized: now 1790577067158734 us; error 36 us; skew 500 ppm
Sep 28 06:31:07 dist-test-slave-d59m krb5kdc[2722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577067, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:07.162811 4053 init.cc:377] Logged in from keytab as kudu/127.22.225.131@KRBTEST.COM (short username kudu)
I20260928 06:31:07.163594 4053 webserver.cc:460] Webserver started at http://127.22.225.131:45075/ using document root <none> and password file <none>
I20260928 06:31:07.163882 4053 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:07.164146 4053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 06:31:07 dist-test-slave-d59m krb5kdc[2722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577067, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.131@KRBTEST.COM for HTTP/127.22.225.148@KRBTEST.COM
I20260928 06:31:07.198971 4053 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:07.200029 4069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:07.200265 4053 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:07.200362 4053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-2/wal
uuid: "73660479c22e4ebe977f06eb4cffab76"
format_stamp: "Formatted at 2026-09-28 06:31:05 on dist-test-slave-d59m"
server_key: "b92a6eeb9740bf0ae8e8fe144d6bde1e"
server_key_iv: "7c3c72fb608e968b377898100159bdd7"
server_key_version: "kuduclusterkey@0"
I20260928 06:31:07.200879 4053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:07.238686 4053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:07.241185 4053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:07.241350 4053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:07.241616 4053 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:31:07.242000 4053 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:31:07.242045 4053 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:07.242101 4053 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:31:07.242131 4053 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:07.249902 4053 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.131:36903
I20260928 06:31:07.249992 4182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.131:36903 every 8 connection(s)
I20260928 06:31:07.250378 4053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
I20260928 06:31:07.257284 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 4053
Sep 28 06:31:07 dist-test-slave-d59m krb5kdc[2722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577067, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.131@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:31:07.266371 4187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:07.257855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57214 (local address 127.22.225.190:43353)
0928 06:31:07.257968 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:07.257970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:07.257979 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:31:07.258004 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:07.258005 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:07.258016 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:07.258089 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:07.258795 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:07.259570 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:07.260548 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:07.260891 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:07.260969 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:07.262436 (+ 1467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:07.262438 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:07.262440 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:07.262450 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:07.264986 (+ 2536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:07.265382 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:07.265383 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:07.265383 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:07.265449 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:07.265673 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:07.265674 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:07.265674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:07.265819 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:07.266019 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:07.266058 (+ 39us) server_negotiation.cc:299] Negotiation successful
0928 06:31:07.266138 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":61,"threads_started":1}
I20260928 06:31:07.268744 3914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:07.256690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:33815 (local address 127.22.225.190:43353)
0928 06:31:07.257642 (+ 952us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:07.257644 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:07.257657 (+ 13us) server_negotiation.cc:407] Connection header received
0928 06:31:07.257689 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:07.257691 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:07.257702 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:07.257776 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:07.258761 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:07.259535 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:07.261105 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:07.261395 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:07.261440 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:07.264567 (+ 3127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:07.264568 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:07.264568 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:07.264592 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:07.267065 (+ 2473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:07.267588 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:07.267590 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:07.267590 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:07.267651 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:07.267958 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:07.267959 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:07.267960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:07.268101 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:07.268147 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:07.268547 (+ 400us) server_negotiation.cc:299] Negotiation successful
0928 06:31:07.268652 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":918}
I20260928 06:31:07.269275 4185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:07.251190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:43353 (local address 127.22.225.131:33815)
0928 06:31:07.251360 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:07.251412 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:07.252070 (+ 658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:07.257910 (+ 5840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:07.257917 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:07.258187 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:07.258626 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:07.258634 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:07.259664 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:07.259665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:07.260993 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:07.260996 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:07.261567 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:07.261568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:07.261700 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:07.262808 (+ 1108us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:07.262818 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:07.264430 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:07.267160 (+ 2730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:07.267167 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:07.267170 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:07.267473 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:07.267733 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:07.267734 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:07.267734 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:07.267875 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:07.268198 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:07.268205 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:07.268368 (+ 163us) client_negotiation.cc:769] Sending connection context
0928 06:31:07.268867 (+ 499us) client_negotiation.cc:240] Negotiation successful
0928 06:31:07.269027 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":80,"threads_started":1}
I20260928 06:31:07.269794 4183 heartbeater.cc:347] Connected to a master server at 127.22.225.190:43353
I20260928 06:31:07.269898 4183 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:07.270082 4183 heartbeater.cc:511] Master 127.22.225.190:43353 requested a full tablet report, sending...
W20260928 06:31:07.270772 4183 heartbeater.cc:644] Failed to heartbeat to 127.22.225.190:43353 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 06:31:07.270840 4183 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:07.270897 4183 heartbeater.cc:511] Master 127.22.225.190:43353 requested a full tablet report, sending...
I20260928 06:31:07.271257 4183 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:07.271327 4183 heartbeater.cc:511] Master 127.22.225.190:43353 requested a full tablet report, sending...
W20260928 06:31:07.271649 4183 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 06:31:07.706719 3911 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:07.706897 3911 heartbeater.cc:511] Master 127.22.225.190:43353 requested a full tablet report, sending...
I20260928 06:31:07.998733 4050 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:07.998904 4050 heartbeater.cc:511] Master 127.22.225.190:43353 requested a full tablet report, sending...
I20260928 06:31:07.999383 3717 ts_manager.cc:194] Registered new tserver with Master: a19efcfb962f45b58fc10d8d60a1fea6 (127.22.225.130:43203)
I20260928 06:31:08.000495 3717 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.225.130@KRBTEST.COM'} at 127.22.225.130:51043
Sep 28 06:31:08 dist-test-slave-d59m krb5kdc[2722](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790577068, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:08.271818 4183 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:08.271987 4183 heartbeater.cc:511] Master 127.22.225.190:43353 requested a full tablet report, sending...
I20260928 06:31:08.272343 3717 ts_manager.cc:194] Registered new tserver with Master: 73660479c22e4ebe977f06eb4cffab76 (127.22.225.131:36903)
I20260928 06:31:08.272897 3717 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.225.131@KRBTEST.COM'} at 127.22.225.131:33815
Sep 28 06:31:08 dist-test-slave-d59m krb5kdc[2722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577068, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.190@KRBTEST.COM for HTTP/127.22.225.148@KRBTEST.COM
I20260928 06:31:08.708695 3911 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:08.708847 3911 heartbeater.cc:511] Master 127.22.225.190:43353 requested a full tablet report, sending...
I20260928 06:31:08.711587 3717 ts_manager.cc:194] Registered new tserver with Master: ccc451df5c534f498bd5fafd19553547 (127.22.225.129:43865)
I20260928 06:31:08.712096 3717 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.225.129@KRBTEST.COM'} at 127.22.225.129:57741
I20260928 06:31:08.716317 23430 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:31:08.730103 4205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:08.717347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57222 (local address 127.22.225.190:43353)
0928 06:31:08.717723 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:08.717725 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:08.717889 (+ 164us) server_negotiation.cc:407] Connection header received
0928 06:31:08.717914 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:08.717914 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:08.717926 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:08.717986 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:08.718736 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:08.719522 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:08.720629 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:08.720886 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:08.720957 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:08.726553 (+ 5596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:08.726555 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:08.726556 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:08.726567 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:08.728531 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:08.728987 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:08.728988 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:08.728989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:08.729057 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:08.729246 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:08.729247 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:08.729247 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:08.729378 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:08.729462 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:08.729910 (+ 448us) server_negotiation.cc:299] Negotiation successful
0928 06:31:08.729998 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":74,"threads_started":1}
I20260928 06:31:08.732321 3717 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57222:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:31:08.734618 3717 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:31:08.746387 4215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:08.738484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.130:43203 (local address 127.0.0.1:53544)
0928 06:31:08.741073 (+ 2589us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:08.741081 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:08.741123 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:08.741387 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:08.741390 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:08.741392 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:08.741467 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:08.741469 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:08.742825 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:08.742826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:08.744326 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:08.744329 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:08.745955 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:08.745956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:08.746152 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:08.746174 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 06:31:08.746244 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 06:31:08.746302 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2574,"thread_start_us":54,"threads_started":1}
I20260928 06:31:08.746419 4216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:08.738746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53544 (local address 127.22.225.130:43203)
0928 06:31:08.738880 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:08.738885 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:08.741136 (+ 2251us) server_negotiation.cc:407] Connection header received
0928 06:31:08.741190 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:08.741193 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:08.741210 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:08.741330 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:08.741541 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:08.742642 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:08.744409 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:08.745172 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:08.745256 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:08.745318 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:08.746280 (+ 962us) server_negotiation.cc:299] Negotiation successful
0928 06:31:08.746330 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":75,"threads_started":1}
I20260928 06:31:08.747449 3984 tablet_service.cc:1467] Processing CreateTablet for tablet c4472a8125a44b74beb9ded1aaef650d (DEFAULT_TABLE table=test-table [id=34c97134cf844084be9c7618a7d888af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:31:08.747845 3984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4472a8125a44b74beb9ded1aaef650d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:08.751541 4211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:08.738108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.129:43865 (local address 127.0.0.1:43826)
0928 06:31:08.738634 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:08.738675 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:08.738742 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:08.741571 (+ 2829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:08.741572 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:08.741573 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:08.741602 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:08.741603 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:08.746665 (+ 5062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:08.746665 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:08.747679 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:08.747682 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:08.751299 (+ 3617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:08.751300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:08.751367 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:08.751372 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:31:08.751405 (+ 33us) client_negotiation.cc:240] Negotiation successful
0928 06:31:08.751450 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":76,"threads_started":1}
I20260928 06:31:08.752008 4212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:08.738201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43826 (local address 127.22.225.129:43865)
0928 06:31:08.741226 (+ 3025us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:08.741232 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:08.741241 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:31:08.741276 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:08.741278 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:08.741294 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:08.741671 (+ 377us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:08.741749 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:08.742737 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:08.747804 (+ 5067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:08.748619 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:08.751754 (+ 3135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:08.751835 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:08.751865 (+ 30us) server_negotiation.cc:299] Negotiation successful
0928 06:31:08.751917 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2990,"thread_start_us":73,"threads_started":1}
I20260928 06:31:08.752796 3844 tablet_service.cc:1467] Processing CreateTablet for tablet c4472a8125a44b74beb9ded1aaef650d (DEFAULT_TABLE table=test-table [id=34c97134cf844084be9c7618a7d888af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:31:08.753167 3844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4472a8125a44b74beb9ded1aaef650d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:08.757359 4213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:08.738337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.131:36903 (local address 127.0.0.1:59066)
0928 06:31:08.740856 (+ 2519us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:08.740866 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:08.740993 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:08.744750 (+ 3757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:08.744751 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:08.744753 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:08.744810 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:08.744812 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:08.748891 (+ 4079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:08.748892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:08.750282 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:08.750285 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:08.752138 (+ 1853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:08.752139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:08.752223 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:08.752227 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:31:08.752257 (+ 30us) client_negotiation.cc:240] Negotiation successful
0928 06:31:08.757212 (+ 4955us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2500,"thread_start_us":61,"threads_started":1}
I20260928 06:31:08.757838 4214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:08.738447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59066 (local address 127.22.225.131:36903)
0928 06:31:08.744486 (+ 6039us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:08.744491 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:08.744502 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:31:08.744536 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:08.744538 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:08.744555 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:08.744658 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:08.744878 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:08.748804 (+ 3926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:08.750396 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:08.751179 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:08.757580 (+ 6401us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:08.757671 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:08.757703 (+ 32us) server_negotiation.cc:299] Negotiation successful
0928 06:31:08.757756 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6001,"thread_start_us":62,"threads_started":1}
I20260928 06:31:08.758602 4117 tablet_service.cc:1467] Processing CreateTablet for tablet c4472a8125a44b74beb9ded1aaef650d (DEFAULT_TABLE table=test-table [id=34c97134cf844084be9c7618a7d888af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:31:08.758970 4117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4472a8125a44b74beb9ded1aaef650d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:08.759202 4217 tablet_bootstrap.cc:492] T c4472a8125a44b74beb9ded1aaef650d P a19efcfb962f45b58fc10d8d60a1fea6: Bootstrap starting.
I20260928 06:31:08.761265 4217 tablet_bootstrap.cc:654] T c4472a8125a44b74beb9ded1aaef650d P a19efcfb962f45b58fc10d8d60a1fea6: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:08.761644 4217 log.cc:824] T c4472a8125a44b74beb9ded1aaef650d P a19efcfb962f45b58fc10d8d60a1fea6: Log is configured to *not* fsync() on all Append() calls
I20260928 06:31:08.764788 4218 tablet_bootstrap.cc:492] T c4472a8125a44b74beb9ded1aaef650d P ccc451df5c534f498bd5fafd19553547: Bootstrap starting.
I20260928 06:31:08.765482 4218 tablet_bootstrap.cc:654] T c4472a8125a44b74beb9ded1aaef650d P ccc451df5c534f498bd5fafd19553547: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:08.765815 4218 log.cc:824] T c4472a8125a44b74beb9ded1aaef650d P ccc451df5c534f498bd5fafd19553547: Log is configured to *not* fsync() on all Append() calls
I20260928 06:31:08.766443 4220 tablet_bootstrap.cc:492] T c4472a8125a44b74beb9ded1aaef650d P 73660479c22e4ebe977f06eb4cffab76: Bootstrap starting.
I20260928 06:31:08.766732 4217 tablet_bootstrap.cc:492] T c4472a8125a44b74beb9ded1aaef650d P a19efcfb962f45b58fc10d8d60a1fea6: No bootstrap required, opened a new log
I20260928 06:31:08.766861 4217 ts_tablet_manager.cc:1397] T c4472a8125a44b74beb9ded1aaef650d P a19efcfb962f45b58fc10d8d60a1fea6: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260928 06:31:08.767112 4220 tablet_bootstrap.cc:654] T c4472a8125a44b74beb9ded1aaef650d P 73660479c22e4ebe977f06eb4cffab76: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:08.767419 4220 log.cc:824] T c4472a8125a44b74beb9ded1aaef650d P 73660479c22e4ebe977f06eb4cffab76: Log is configured to *not* fsync() on all Append() calls
I20260928 06:31:08.768285 4218 tablet_bootstrap.cc:492] T c4472a8125a44b74beb9ded1aaef650d P ccc451df5c534f498bd5fafd19553547: No bootstrap required, opened a new log
I20260928 06:31:08.768360 4218 ts_tablet_manager.cc:1397] T c4472a8125a44b74beb9ded1aaef650d P ccc451df5c534f498bd5fafd19553547: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260928 06:31:08.770152 4217 raft_consensus.cc:348] T c4472a8125a44b74beb9ded1aaef650d P a19efcfb962f45b58fc10d8d60a1fea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccc451df5c534f498bd5fafd19553547" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 43865 } } peers { permanent_uuid: "73660479c22e4ebe977f06eb4cffab76" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 36903 } } peers { permanent_uuid: "a19efcfb962f45b58fc10d8d60a1fea6" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 43203 } }
I20260928 06:31:08.770427 4217 raft_consensus.cc:374] T c4472a8125a44b74beb9ded1aaef650d P a19efcfb962f45b58fc10d8d60a1fea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:08.770515 4217 raft_consensus.cc:729] T c4472a8125a44b74beb9ded1aaef650d P a19efcfb962f45b58fc10d8d60a1fea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a19efcfb962f45b58fc10d8d60a1fea6, State: Initialized, Role: FOLLOWER
I20260928 06:31:08.770735 4217 consensus_queue.cc:260] T c4472a8125a44b74beb9ded1aaef650d P a19efcfb962f45b58fc10d8d60a1fea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccc451df5c534f498bd5fafd19553547" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 43865 } } peers { permanent_uuid: "73660479c22e4ebe977f06eb4cffab76" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 36903 } } peers { permanent_uuid: "a19efcfb962f45b58fc10d8d60a1fea6" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 43203 } }
I20260928 06:31:08.771118 4217 ts_tablet_manager.cc:1428] T c4472a8125a44b74beb9ded1aaef650d P a19efcfb962f45b58fc10d8d60a1fea6: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260928 06:31:08.771461 4218 raft_consensus.cc:348] T c4472a8125a44b74beb9ded1aaef650d P ccc451df5c534f498bd5fafd19553547 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccc451df5c534f498bd5fafd19553547" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 43865 } } peers { permanent_uuid: "73660479c22e4ebe977f06eb4cffab76" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 36903 } } peers { permanent_uuid: "a19efcfb962f45b58fc10d8d60a1fea6" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 43203 } }
I20260928 06:31:08.771696 4218 raft_consensus.cc:374] T c4472a8125a44b74beb9ded1aaef650d P ccc451df5c534f498bd5fafd19553547 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:08.771744 4218 raft_consensus.cc:729] T c4472a8125a44b74beb9ded1aaef650d P ccc451df5c534f498bd5fafd19553547 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccc451df5c534f498bd5fafd19553547, State: Initialized, Role: FOLLOWER
I20260928 06:31:08.771899 4050 heartbeater.cc:503] Master 127.22.225.190:43353 was elected leader, sending a full tablet report...
I20260928 06:31:08.771888 4218 consensus_queue.cc:260] T c4472a8125a44b74beb9ded1aaef650d P ccc451df5c534f498bd5fafd19553547 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccc451df5c534f498bd5fafd19553547" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 43865 } } peers { permanent_uuid: "73660479c22e4ebe977f06eb4cffab76" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 36903 } } peers { permanent_uuid: "a19efcfb962f45b58fc10d8d60a1fea6" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 43203 } }
I20260928 06:31:08.772147 4218 ts_tablet_manager.cc:1428] T c4472a8125a44b74beb9ded1aaef650d P ccc451df5c534f498bd5fafd19553547: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260928 06:31:08.772255 3911 heartbeater.cc:503] Master 127.22.225.190:43353 was elected leader, sending a full tablet report...
I20260928 06:31:08.773952 4220 tablet_bootstrap.cc:492] T c4472a8125a44b74beb9ded1aaef650d P 73660479c22e4ebe977f06eb4cffab76: No bootstrap required, opened a new log
I20260928 06:31:08.774036 4220 ts_tablet_manager.cc:1397] T c4472a8125a44b74beb9ded1aaef650d P 73660479c22e4ebe977f06eb4cffab76: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260928 06:31:08.778115 4220 raft_consensus.cc:348] T c4472a8125a44b74beb9ded1aaef650d P 73660479c22e4ebe977f06eb4cffab76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccc451df5c534f498bd5fafd19553547" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 43865 } } peers { permanent_uuid: "73660479c22e4ebe977f06eb4cffab76" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 36903 } } peers { permanent_uuid: "a19efcfb962f45b58fc10d8d60a1fea6" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 43203 } }
I20260928 06:31:08.778309 4220 raft_consensus.cc:374] T c4472a8125a44b74beb9ded1aaef650d P 73660479c22e4ebe977f06eb4cffab76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:08.778352 4220 raft_consensus.cc:729] T c4472a8125a44b74beb9ded1aaef650d P 73660479c22e4ebe977f06eb4cffab76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73660479c22e4ebe977f06eb4cffab76, State: Initialized, Role: FOLLOWER
I20260928 06:31:08.778481 4220 consensus_queue.cc:260] T c4472a8125a44b74beb9ded1aaef650d P 73660479c22e4ebe977f06eb4cffab76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccc451df5c534f498bd5fafd19553547" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 43865 } } peers { permanent_uuid: "73660479c22e4ebe977f06eb4cffab76" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 36903 } } peers { permanent_uuid: "a19efcfb962f45b58fc10d8d60a1fea6" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 43203 } }
I20260928 06:31:08.778929 4183 heartbeater.cc:503] Master 127.22.225.190:43353 was elected leader, sending a full tablet report...
I20260928 06:31:08.779141 4220 ts_tablet_manager.cc:1428] T c4472a8125a44b74beb9ded1aaef650d P 73660479c22e4ebe977f06eb4cffab76: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260928 06:31:08.936138 3912 tablet.cc:2369] T c4472a8125a44b74beb9ded1aaef650d P ccc451df5c534f498bd5fafd19553547: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:31:08.974953 4051 tablet.cc:2369] T c4472a8125a44b74beb9ded1aaef650d P a19efcfb962f45b58fc10d8d60a1fea6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:31:09.002033 4184 tablet.cc:2369] T c4472a8125a44b74beb9ded1aaef650d P 73660479c22e4ebe977f06eb4cffab76: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 06:31:09 dist-test-slave-d59m krb5kdc[2722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577068, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.190@KRBTEST.COM for HTTP/127.22.225.148@KRBTEST.COM
I20260928 06:31:09.138991 4225 raft_consensus.cc:482] T c4472a8125a44b74beb9ded1aaef650d P 73660479c22e4ebe977f06eb4cffab76 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:31:09.139160 4225 raft_consensus.cc:504] T c4472a8125a44b74beb9ded1aaef650d P 73660479c22e4ebe977f06eb4cffab76 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccc451df5c534f498bd5fafd19553547" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 43865 } } peers { permanent_uuid: "73660479c22e4ebe977f06eb4cffab76" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 36903 } } peers { permanent_uuid: "a19efcfb962f45b58fc10d8d60a1fea6" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 43203 } }
I20260928 06:31:09.139642 4225 leader_election.cc:288] T c4472a8125a44b74beb9ded1aaef650d P 73660479c22e4ebe977f06eb4cffab76 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ccc451df5c534f498bd5fafd19553547 (127.22.225.129:43865), a19efcfb962f45b58fc10d8d60a1fea6 (127.22.225.130:43203)
I20260928 06:31:09.151250 4212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:09.140034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:49565 (local address 127.22.225.129:43865)
0928 06:31:09.140092 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:09.140095 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:09.141616 (+ 1521us) server_negotiation.cc:407] Connection header received
0928 06:31:09.145044 (+ 3428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:09.145046 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:09.145060 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:09.145095 (+ 35us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:09.145963 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:09.146627 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:09.150187 (+ 3560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:09.150801 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:09.150856 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:09.150873 (+ 17us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:09.151117 (+ 244us) server_negotiation.cc:299] Negotiation successful
0928 06:31:09.151166 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 06:31:09.151786 4228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:09.139969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.129:43865 (local address 127.22.225.131:49565)
0928 06:31:09.141533 (+ 1564us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:09.141548 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:09.141659 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:09.145180 (+ 3521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:09.145181 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:09.145184 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:09.145274 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:09.145276 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:09.147147 (+ 1871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:09.147148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:09.148483 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:09.148487 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:09.150956 (+ 2469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:09.150957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:09.151034 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:09.151038 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:31:09.151611 (+ 573us) client_negotiation.cc:240] Negotiation successful
0928 06:31:09.151675 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1544,"thread_start_us":108,"threads_started":1}
I20260928 06:31:09.152248 4216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:09.140494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:33473 (local address 127.22.225.130:43203)
0928 06:31:09.140538 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:09.140540 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:09.148751 (+ 8211us) server_negotiation.cc:407] Connection header received
0928 06:31:09.148816 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:09.148818 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:09.148831 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:09.148899 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:09.149155 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:09.149892 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:09.151253 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:09.151838 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:09.151889 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:09.151896 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:09.152133 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 06:31:09.152177 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:31:09.152302 3865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4472a8125a44b74beb9ded1aaef650d" candidate_uuid: "73660479c22e4ebe977f06eb4cffab76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccc451df5c534f498bd5fafd19553547" is_pre_election: true
I20260928 06:31:09.152472 3865 raft_consensus.cc:2454] T c4472a8125a44b74beb9ded1aaef650d P ccc451df5c534f498bd5fafd19553547 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73660479c22e4ebe977f06eb4cffab76 in term 0.
I20260928 06:31:09.152513 4229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:09.140317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.130:43203 (local address 127.22.225.131:33473)
0928 06:31:09.148668 (+ 8351us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:09.148683 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:09.148754 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:09.148956 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:09.148958 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:09.148960 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:09.149054 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:09.149056 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:09.149998 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:09.149999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:09.151143 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:09.151146 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:09.151970 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:09.151971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:09.152046 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:09.152049 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:31:09.152372 (+ 323us) client_negotiation.cc:240] Negotiation successful
0928 06:31:09.152420 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8328,"thread_start_us":63,"threads_started":1}
I20260928 06:31:09.152815 4004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4472a8125a44b74beb9ded1aaef650d" candidate_uuid: "73660479c22e4ebe977f06eb4cffab76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a19efcfb962f45b58fc10d8d60a1fea6" is_pre_election: true
I20260928 06:31:09.152966 4004 raft_consensus.cc:2454] T c4472a8125a44b74beb9ded1aaef650d P a19efcfb962f45b58fc10d8d60a1fea6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73660479c22e4ebe977f06eb4cffab76 in term 0.
I20260928 06:31:09.153081 4070 leader_election.cc:302] T c4472a8125a44b74beb9ded1aaef650d P 73660479c22e4ebe977f06eb4cffab76 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73660479c22e4ebe977f06eb4cffab76, ccc451df5c534f498bd5fafd19553547; no voters:
I20260928 06:31:09.153324 4225 raft_consensus.cc:2781] T c4472a8125a44b74beb9ded1aaef650d P 73660479c22e4ebe977f06eb4cffab76 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:31:09.153381 4225 raft_consensus.cc:482] T c4472a8125a44b74beb9ded1aaef650d P 73660479c22e4ebe977f06eb4cffab76 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:31:09.153404 4225 raft_consensus.cc:3037] T c4472a8125a44b74beb9ded1aaef650d P 73660479c22e4ebe977f06eb4cffab76 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:09.154383 4225 raft_consensus.cc:504] T c4472a8125a44b74beb9ded1aaef650d P 73660479c22e4ebe977f06eb4cffab76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccc451df5c534f498bd5fafd19553547" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 43865 } } peers { permanent_uuid: "73660479c22e4ebe977f06eb4cffab76" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 36903 } } peers { permanent_uuid: "a19efcfb962f45b58fc10d8d60a1fea6" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 43203 } }
I20260928 06:31:09.154525 4225 leader_election.cc:288] T c4472a8125a44b74beb9ded1aaef650d P 73660479c22e4ebe977f06eb4cffab76 [CANDIDATE]: Term 1 election: Requested vote from peers ccc451df5c534f498bd5fafd19553547 (127.22.225.129:43865), a19efcfb962f45b58fc10d8d60a1fea6 (127.22.225.130:43203)
I20260928 06:31:09.154688 4004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4472a8125a44b74beb9ded1aaef650d" candidate_uuid: "73660479c22e4ebe977f06eb4cffab76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a19efcfb962f45b58fc10d8d60a1fea6"
I20260928 06:31:09.154778 4004 raft_consensus.cc:3037] T c4472a8125a44b74beb9ded1aaef650d P a19efcfb962f45b58fc10d8d60a1fea6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:09.154917 3865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4472a8125a44b74beb9ded1aaef650d" candidate_uuid: "73660479c22e4ebe977f06eb4cffab76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccc451df5c534f498bd5fafd19553547"
I20260928 06:31:09.155006 3865 raft_consensus.cc:3037] T c4472a8125a44b74beb9ded1aaef650d P ccc451df5c534f498bd5fafd19553547 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:09.155680 4004 raft_consensus.cc:2454] T c4472a8125a44b74beb9ded1aaef650d P a19efcfb962f45b58fc10d8d60a1fea6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73660479c22e4ebe977f06eb4cffab76 in term 1.
I20260928 06:31:09.155886 3865 raft_consensus.cc:2454] T c4472a8125a44b74beb9ded1aaef650d P ccc451df5c534f498bd5fafd19553547 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73660479c22e4ebe977f06eb4cffab76 in term 1.
I20260928 06:31:09.155893 4073 leader_election.cc:302] T c4472a8125a44b74beb9ded1aaef650d P 73660479c22e4ebe977f06eb4cffab76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73660479c22e4ebe977f06eb4cffab76, a19efcfb962f45b58fc10d8d60a1fea6; no voters:
I20260928 06:31:09.156180 4225 raft_consensus.cc:2781] T c4472a8125a44b74beb9ded1aaef650d P 73660479c22e4ebe977f06eb4cffab76 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:31:09.156354 4225 raft_consensus.cc:686] T c4472a8125a44b74beb9ded1aaef650d P 73660479c22e4ebe977f06eb4cffab76 [term 1 LEADER]: Becoming Leader. State: Replica: 73660479c22e4ebe977f06eb4cffab76, State: Running, Role: LEADER
I20260928 06:31:09.156448 4225 consensus_queue.cc:237] T c4472a8125a44b74beb9ded1aaef650d P 73660479c22e4ebe977f06eb4cffab76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccc451df5c534f498bd5fafd19553547" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 43865 } } peers { permanent_uuid: "73660479c22e4ebe977f06eb4cffab76" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 36903 } } peers { permanent_uuid: "a19efcfb962f45b58fc10d8d60a1fea6" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 43203 } }
I20260928 06:31:09.157382 3718 catalog_manager.cc:5495] T c4472a8125a44b74beb9ded1aaef650d P 73660479c22e4ebe977f06eb4cffab76 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73660479c22e4ebe977f06eb4cffab76 (127.22.225.131). New cstate: current_term: 1 leader_uuid: "73660479c22e4ebe977f06eb4cffab76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccc451df5c534f498bd5fafd19553547" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73660479c22e4ebe977f06eb4cffab76" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 36903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a19efcfb962f45b58fc10d8d60a1fea6" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 43203 } health_report { overall_health: UNKNOWN } } }
I20260928 06:31:09.185706 4214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:09.182513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59072 (local address 127.22.225.131:36903)
0928 06:31:09.182631 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:09.182633 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:09.182641 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:31:09.182683 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:09.182684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:09.182705 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:09.182783 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:31:09.182984 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:09.183631 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:09.184662 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:09.184914 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:09.184970 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:09.185162 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:31:09.185291 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:31:09.185358 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:09.185605 (+ 247us) server_negotiation.cc:299] Negotiation successful
0928 06:31:09.185626 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260928 06:31:09.188138 4004 raft_consensus.cc:1261] T c4472a8125a44b74beb9ded1aaef650d P a19efcfb962f45b58fc10d8d60a1fea6 [term 1 FOLLOWER]: Refusing update from remote peer 73660479c22e4ebe977f06eb4cffab76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:31:09.188632 4230 consensus_queue.cc:1035] T c4472a8125a44b74beb9ded1aaef650d P 73660479c22e4ebe977f06eb4cffab76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a19efcfb962f45b58fc10d8d60a1fea6" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 43203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:31:09.190923 3865 raft_consensus.cc:1261] T c4472a8125a44b74beb9ded1aaef650d P ccc451df5c534f498bd5fafd19553547 [term 1 FOLLOWER]: Refusing update from remote peer 73660479c22e4ebe977f06eb4cffab76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:31:09.191289 4230 consensus_queue.cc:1035] T c4472a8125a44b74beb9ded1aaef650d P 73660479c22e4ebe977f06eb4cffab76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccc451df5c534f498bd5fafd19553547" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:31:09.200155 4233 mvcc.cc:205] Tried to move back new op lower bound from 7334203675391352832 to 7334203675265437696. Current Snapshot: MvccSnapshot[applied={T|T < 7334203675391352832 or (T in {7334203675391352832})}]
I20260928 06:31:09.201691 3718 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:31:09.201804 3718 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:31:09.202888 3718 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fb7bf101bf764f27960d5a5ad1825a73: Sending DeleteTablet for 3 replicas of tablet c4472a8125a44b74beb9ded1aaef650d
I20260928 06:31:09.203544 3844 tablet_service.cc:1514] Processing DeleteTablet for tablet c4472a8125a44b74beb9ded1aaef650d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:31:09 UTC) from {username='kudu', principal='kudu/127.22.225.190@KRBTEST.COM'} at 127.0.0.1:43826
I20260928 06:31:09.203801 4117 tablet_service.cc:1514] Processing DeleteTablet for tablet c4472a8125a44b74beb9ded1aaef650d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:31:09 UTC) from {username='kudu', principal='kudu/127.22.225.190@KRBTEST.COM'} at 127.0.0.1:59066
I20260928 06:31:09.204025 3984 tablet_service.cc:1514] Processing DeleteTablet for tablet c4472a8125a44b74beb9ded1aaef650d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:31:09 UTC) from {username='kudu', principal='kudu/127.22.225.190@KRBTEST.COM'} at 127.0.0.1:53544
I20260928 06:31:09.204939 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 3754
I20260928 06:31:09.207343 4234 mvcc.cc:205] Tried to move back new op lower bound from 7334203675391352832 to 7334203675265437696. Current Snapshot: MvccSnapshot[applied={T|T < 7334203675391352832 or (T in {7334203675391352832})}]
I20260928 06:31:09.208972 4242 tablet_replica.cc:330] T c4472a8125a44b74beb9ded1aaef650d P 73660479c22e4ebe977f06eb4cffab76: stopping tablet replica
I20260928 06:31:09.209180 4243 tablet_replica.cc:330] T c4472a8125a44b74beb9ded1aaef650d P a19efcfb962f45b58fc10d8d60a1fea6: stopping tablet replica
I20260928 06:31:09.210803 4243 raft_consensus.cc:2229] T c4472a8125a44b74beb9ded1aaef650d P a19efcfb962f45b58fc10d8d60a1fea6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:31:09.211020 4242 raft_consensus.cc:2229] T c4472a8125a44b74beb9ded1aaef650d P 73660479c22e4ebe977f06eb4cffab76 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:31:09.211203 4242 raft_consensus.cc:2258] T c4472a8125a44b74beb9ded1aaef650d P 73660479c22e4ebe977f06eb4cffab76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:31:09.211391 4243 raft_consensus.cc:2258] T c4472a8125a44b74beb9ded1aaef650d P a19efcfb962f45b58fc10d8d60a1fea6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:31:09.212086 4242 ts_tablet_manager.cc:1905] T c4472a8125a44b74beb9ded1aaef650d P 73660479c22e4ebe977f06eb4cffab76: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:31:09.212631 4243 ts_tablet_manager.cc:1905] T c4472a8125a44b74beb9ded1aaef650d P a19efcfb962f45b58fc10d8d60a1fea6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 06:31:09.213462 4070 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.225.129:43865 (error 108)
W20260928 06:31:09.213967 3702 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.225.129:43865 (error 108)
I20260928 06:31:09.214756 4242 ts_tablet_manager.cc:1918] T c4472a8125a44b74beb9ded1aaef650d P 73660479c22e4ebe977f06eb4cffab76: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:31:09.214982 4213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:09.214831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.129:43865 (local address 127.0.0.1:43828)
0928 06:31:09.214902 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:09.214945 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.129:43865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 06:31:09.215114 3702 catalog_manager.cc:4584] TS ccc451df5c534f498bd5fafd19553547 (127.22.225.129:43865): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c4472a8125a44b74beb9ded1aaef650d: Network error: Client connection negotiation failed: client connection to 127.22.225.129:43865: connect: Connection refused (error 111)
I20260928 06:31:09.215355 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 3916
I20260928 06:31:09.219446 4228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:09.219315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.129:43865 (local address 127.22.225.131:38529)
0928 06:31:09.219364 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:09.219409 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.129:43865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 06:31:09.224723 4242 log.cc:1196] T c4472a8125a44b74beb9ded1aaef650d P 73660479c22e4ebe977f06eb4cffab76: Deleting WAL directory at /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576899143097-23430-0/minicluster-data/ts-2/wal/wals/c4472a8125a44b74beb9ded1aaef650d
I20260928 06:31:09.225106 4242 ts_tablet_manager.cc:1939] T c4472a8125a44b74beb9ded1aaef650d P 73660479c22e4ebe977f06eb4cffab76: Deleting consensus metadata
I20260928 06:31:09.225698 3704 catalog_manager.cc:4841] TS 73660479c22e4ebe977f06eb4cffab76 (127.22.225.131:36903): tablet c4472a8125a44b74beb9ded1aaef650d (table test-table [id=34c97134cf844084be9c7618a7d888af]) successfully deleted
W20260928 06:31:09.232374 4213 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:31:09.232006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.130:43203 (local address 127.0.0.1:53546)
0928 06:31:09.232070 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:09.232083 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:09.232140 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:09.232310 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.130:43203: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 06:31:09.236785 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 4053
I20260928 06:31:09.243541 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 3685
I20260928 06:31:09.251554 23430 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 06:31:09.596326 23430 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 06:31:09.596433 23430 mini_ranger.cc:66] Stopping Ranger...
I20260928 06:31:09.941318 23430 mini_ranger.cc:68] Stopped Ranger
2026-09-28 06:31:09.941 UTC [2752] LOG: received smart shutdown request
2026-09-28 06:31:09.945 UTC [2752] LOG: background worker "logical replication launcher" (PID 2761) exited with exit code 1
2026-09-28 06:31:09.945 UTC [2756] LOG: shutting down
2026-09-28 06:31:10.055 UTC [2752] LOG: database system is shut down
2026-09-28T06:31:10Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (55970 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:31:10 dist-test-slave-d59m krb5kdc[4254](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:31:10 dist-test-slave-d59m krb5kdc[4254](info): set up 2 sockets
Sep 28 06:31:10 dist-test-slave-d59m krb5kdc[4254](info): commencing operation
krb5kdc: starting...
W20260928 06:31:11.238494 23430 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Sep 28 06:31:11 dist-test-slave-d59m krb5kdc[4254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:31:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:31:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab.
Entry for principal kudu/127.22.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab.
I20260928 06:31:11.360914 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:36075
--webserver_interface=127.22.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab
--principal=kudu/127.22.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:44143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.225.190:36075,127.22.225.189:33341,127.22.225.188:36225
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:31:11.448714 4267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:11.448913 4267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:11.448938 4267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:11.450249 4267 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:31:11.450292 4267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:11.450307 4267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:11.450320 4267 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:31:11.450332 4267 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:31:11.451843 4267 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:44143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.225.190:36075,127.22.225.189:33341,127.22.225.188:36225
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab
--principal=kudu/127.22.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:36075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4267
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:31:11.452100 4267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:11.452374 4267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:11.457417 4275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:11.457418 4272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:11.457422 4273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:11.457590 4267 server_base.cc:1034] running on GCE node
I20260928 06:31:11.458045 4267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:11.458420 4267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:11.459609 4267 hybrid_clock.cc:648] HybridClock initialized: now 1790577071459595 us; error 31 us; skew 500 ppm
Sep 28 06:31:11 dist-test-slave-d59m krb5kdc[4254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577071, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:11.462602 4267 init.cc:377] Logged in from keytab as kudu/127.22.225.190@KRBTEST.COM (short username kudu)
I20260928 06:31:11.463045 4267 webserver.cc:460] Webserver started at http://127.22.225.190:38433/ using document root <none> and password file <none>
I20260928 06:31:11.463284 4267 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:11.463353 4267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:11.463507 4267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:11.464695 4267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/minicluster-data/master-0/data/instance:
uuid: "1bd1e649645a4d5ca66a6e9a7dea54b3"
format_stamp: "Formatted at 2026-09-28 06:31:11 on dist-test-slave-d59m"
server_key: "c51c7c312f71c9904140aeb76b74df02"
server_key_iv: "6da2cbdf9182dd22f43af82e1b41ab5f"
server_key_version: "encryptionkey@0"
I20260928 06:31:11.465057 4267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/minicluster-data/master-0/wal/instance:
uuid: "1bd1e649645a4d5ca66a6e9a7dea54b3"
format_stamp: "Formatted at 2026-09-28 06:31:11 on dist-test-slave-d59m"
server_key: "c51c7c312f71c9904140aeb76b74df02"
server_key_iv: "6da2cbdf9182dd22f43af82e1b41ab5f"
server_key_version: "encryptionkey@0"
I20260928 06:31:11.466547 4267 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:11.467329 4282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:11.467509 4267 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:11.467610 4267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/minicluster-data/master-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/minicluster-data/master-0/wal
uuid: "1bd1e649645a4d5ca66a6e9a7dea54b3"
format_stamp: "Formatted at 2026-09-28 06:31:11 on dist-test-slave-d59m"
server_key: "c51c7c312f71c9904140aeb76b74df02"
server_key_iv: "6da2cbdf9182dd22f43af82e1b41ab5f"
server_key_version: "encryptionkey@0"
I20260928 06:31:11.467692 4267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:11.477629 4267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:11.479933 4267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:11.480109 4267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:11.484834 4267 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.190:36075
I20260928 06:31:11.484879 4334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.190:36075 every 8 connection(s)
I20260928 06:31:11.485248 4267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
I20260928 06:31:11.485947 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 4267
I20260928 06:31:11.486052 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/minicluster-data/master-0/wal/instance
I20260928 06:31:11.486228 4335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:11.486260 23430 external_mini_cluster.cc:1428] Setting key ef36561b055be3ba6b6a849d415ef528
I20260928 06:31:11.488229 4335 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36075 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.22.225.189@KRBTEST.COM; defaulting to no policy
Sep 28 06:31:11 dist-test-slave-d59m krb5kdc[4254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577071, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.190@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.189@KRBTEST.COM" created.
I20260928 06:31:11.499058 4339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:11.489004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36075 (local address 127.0.0.1:39676)
0928 06:31:11.489206 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:11.489249 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:11.489979 (+ 730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:11.490328 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:11.490331 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:11.490619 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:11.490776 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:11.490781 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:11.491836 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:11.491837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:11.492515 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:11.492517 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:11.493109 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:11.493110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:11.493180 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:11.493733 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:11.493742 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:11.495101 (+ 1359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:11.497186 (+ 2085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:11.497194 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:11.497196 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:11.497463 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:11.497803 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:11.497804 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:11.497805 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:11.497920 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:11.498415 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:11.498423 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:11.498503 (+ 80us) client_negotiation.cc:769] Sending connection context
0928 06:31:11.498607 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 06:31:11.498779 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":117,"threads_started":1}
I20260928 06:31:11.499064 4338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:11.489011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39676 (local address 127.22.225.190:36075)
0928 06:31:11.489200 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:11.489209 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:11.489332 (+ 123us) server_negotiation.cc:407] Connection header received
0928 06:31:11.490183 (+ 851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:11.490185 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:11.490209 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:11.490322 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:11.490888 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:11.491737 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:11.492676 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:11.493009 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:11.493140 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:11.495213 (+ 2073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:11.495221 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:11.495221 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:11.495227 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:11.497076 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:11.497562 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:11.497565 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:11.497565 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:11.497712 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:11.498010 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:11.498011 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:11.498011 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:11.498327 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:11.498415 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:11.498657 (+ 242us) server_negotiation.cc:299] Negotiation successful
0928 06:31:11.498884 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":90,"server-negotiator.queue_time_us":140,"thread_start_us":91,"threads_started":1}
I20260928 06:31:11.500146 4335 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.225.189" port: 33341 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:31:11.500523 4339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:11.500396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.189:33341 (local address 127.0.0.1:34272)
0928 06:31:11.500450 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:11.500494 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.189:33341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 06:31:11.500720 4285 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.225.189:33341: connect: Connection refused (error 111)
I20260928 06:31:11.501134 4339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:11.501042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.189:33341 (local address 127.0.0.1:34274)
0928 06:31:11.501088 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:11.501113 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.189:33341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 06:31:11.501330 4335 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.225.189:33341: Network error: Client connection negotiation failed: client connection to 127.22.225.189:33341: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.189.keytab.
Entry for principal kudu/127.22.225.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.189.keytab.
I20260928 06:31:11.513779 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.189:33341
--webserver_interface=127.22.225.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.189.keytab
--principal=kudu/127.22.225.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:44143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.225.190:36075,127.22.225.189:33341,127.22.225.188:36225
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:31:11.550664 4335 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.225.189" port: 33341 } attempt: 1
I20260928 06:31:11.551206 4339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:11.551064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.189:33341 (local address 127.0.0.1:34278)
0928 06:31:11.551135 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:11.551176 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.189:33341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 06:31:11.551592 4339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:11.551490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.189:33341 (local address 127.0.0.1:34292)
0928 06:31:11.551540 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:11.551573 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.189:33341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 06:31:11.551779 4335 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.225.189:33341: Network error: Client connection negotiation failed: client connection to 127.22.225.189:33341: connect: Connection refused (error 111)
W20260928 06:31:11.614362 4342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:11.614575 4342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:11.614598 4342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:11.615787 4342 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:31:11.615832 4342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:11.615846 4342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:11.615859 4342 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:31:11.615871 4342 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:31:11.617478 4342 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:44143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.22.225.190:36075,127.22.225.189:33341,127.22.225.188:36225
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.189.keytab
--principal=kudu/127.22.225.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.189:33341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.22.225.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:31:11.617746 4342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:11.617988 4342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:31:11.619937 4335 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.225.189" port: 33341 } attempt: 2
I20260928 06:31:11.620411 4339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:11.620270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.189:33341 (local address 127.0.0.1:34294)
0928 06:31:11.620336 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:11.620383 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.189:33341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 06:31:11.620813 4339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:11.620712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.189:33341 (local address 127.0.0.1:34310)
0928 06:31:11.620762 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:11.620794 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.189:33341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 06:31:11.620918 4335 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.225.189:33341: Network error: Client connection negotiation failed: client connection to 127.22.225.189:33341: connect: Connection refused (error 111)
W20260928 06:31:11.624145 4347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:11.624121 4348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:11.624109 4350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:11.624672 4342 server_base.cc:1034] running on GCE node
I20260928 06:31:11.624964 4342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:11.625322 4342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:11.626524 4342 hybrid_clock.cc:648] HybridClock initialized: now 1790577071626497 us; error 42 us; skew 500 ppm
Sep 28 06:31:11 dist-test-slave-d59m krb5kdc[4254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577071, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:11.629760 4342 init.cc:377] Logged in from keytab as kudu/127.22.225.189@KRBTEST.COM (short username kudu)
I20260928 06:31:11.630285 4342 webserver.cc:460] Webserver started at http://127.22.225.189:39423/ using document root <none> and password file <none>
I20260928 06:31:11.630632 4342 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:11.630713 4342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:11.630880 4342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:11.632381 4342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/minicluster-data/master-1/data/instance:
uuid: "6c16feb28f634d35adeb6b39e97b2409"
format_stamp: "Formatted at 2026-09-28 06:31:11 on dist-test-slave-d59m"
server_key: "de5561e8e100ff3c367ac45854fc75a0"
server_key_iv: "a3508f45899a9cd03ae532cd32d55b06"
server_key_version: "encryptionkey@0"
I20260928 06:31:11.632890 4342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/minicluster-data/master-1/wal/instance:
uuid: "6c16feb28f634d35adeb6b39e97b2409"
format_stamp: "Formatted at 2026-09-28 06:31:11 on dist-test-slave-d59m"
server_key: "de5561e8e100ff3c367ac45854fc75a0"
server_key_iv: "a3508f45899a9cd03ae532cd32d55b06"
server_key_version: "encryptionkey@0"
I20260928 06:31:11.634862 4342 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260928 06:31:11.635835 4357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:11.636035 4342 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:11.636145 4342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/minicluster-data/master-1/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/minicluster-data/master-1/wal
uuid: "6c16feb28f634d35adeb6b39e97b2409"
format_stamp: "Formatted at 2026-09-28 06:31:11 on dist-test-slave-d59m"
server_key: "de5561e8e100ff3c367ac45854fc75a0"
server_key_iv: "a3508f45899a9cd03ae532cd32d55b06"
server_key_version: "encryptionkey@0"
I20260928 06:31:11.636224 4342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:11.668131 4342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:11.670576 4342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:11.670739 4342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:11.675312 4342 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.189:33341
I20260928 06:31:11.675367 4409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.189:33341 every 8 connection(s)
I20260928 06:31:11.675742 4342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/minicluster-data/master-1/data/info.pb
I20260928 06:31:11.676824 4410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:11.679028 4410 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36075 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:31:11.679848 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 4342
I20260928 06:31:11.679953 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/minicluster-data/master-1/wal/instance
I20260928 06:31:11.680158 23430 external_mini_cluster.cc:1428] Setting key f47f4bc2cb2ad5161c50ee727ed65f8a
Sep 28 06:31:11 dist-test-slave-d59m krb5kdc[4254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577071, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.189@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.225.188@KRBTEST.COM; defaulting to no policy
I20260928 06:31:11.690416 4338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:11.679865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39688 (local address 127.22.225.190:36075)
0928 06:31:11.679918 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:11.679921 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:11.680216 (+ 295us) server_negotiation.cc:407] Connection header received
0928 06:31:11.680844 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:11.680846 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:11.680864 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:11.680921 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:11.681678 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:11.682482 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:11.683527 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:11.683882 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:11.683952 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:11.687085 (+ 3133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:11.687086 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:11.687087 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:11.687095 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:11.688910 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:11.689423 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:11.689424 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:11.689425 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:11.689490 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:11.689746 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:11.689746 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:11.689747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:11.689894 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:11.689968 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:11.690221 (+ 253us) server_negotiation.cc:299] Negotiation successful
0928 06:31:11.690312 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 06:31:11.690685 4412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:11.679848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36075 (local address 127.0.0.1:39688)
0928 06:31:11.680080 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:11.680127 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:11.680715 (+ 588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:11.681002 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:11.681006 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:11.681290 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:11.681545 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:11.681552 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:11.682616 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:11.682617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:11.683429 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:11.683431 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:11.684037 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:11.684039 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:11.684200 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:11.684934 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:11.684945 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:11.686971 (+ 2026us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:11.689003 (+ 2032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:11.689011 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:11.689013 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:11.689340 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:11.689570 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:11.689571 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:11.689571 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:11.689678 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:11.689963 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:11.689967 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:11.690091 (+ 124us) client_negotiation.cc:769] Sending connection context
0928 06:31:11.690177 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 06:31:11.690385 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":87,"threads_started":1}
I20260928 06:31:11.691315 4410 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.225.189" port: 33341 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.188@KRBTEST.COM" created.
Sep 28 06:31:11 dist-test-slave-d59m krb5kdc[4254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577071, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.189@KRBTEST.COM for kudu/127.22.225.189@KRBTEST.COM
I20260928 06:31:11.699635 4412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:11.691596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.189:33341 (local address 127.0.0.1:34316)
0928 06:31:11.691649 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:11.691659 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:11.691720 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:11.692000 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:11.692001 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:11.692150 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:11.692220 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:11.692222 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:11.693673 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:11.693675 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:11.694505 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:11.694508 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:11.694971 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:11.694972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:11.695047 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:11.695485 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:11.695490 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:11.696723 (+ 1233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:11.698533 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:11.698535 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:11.698535 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:11.698704 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:11.698917 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:11.698918 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:11.698919 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:11.698973 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:11.699270 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:11.699271 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:11.699311 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 06:31:11.699406 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 06:31:11.699506 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 06:31:11.699664 4414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:11.691683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34316 (local address 127.22.225.189:33341)
0928 06:31:11.691859 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:11.691862 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:11.691878 (+ 16us) server_negotiation.cc:407] Connection header received
0928 06:31:11.691913 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:11.691915 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:11.691929 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:11.692000 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:11.692318 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:11.693521 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:11.694588 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:11.694894 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:11.694975 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:11.696807 (+ 1832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:11.696815 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:11.696816 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:11.696822 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:11.698428 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:11.698793 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:11.698796 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:11.698796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:11.698851 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:11.699052 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:11.699053 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:11.699053 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:11.699193 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:11.699280 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:11.699438 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 06:31:11.699558 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":151,"thread_start_us":116,"threads_started":1}
I20260928 06:31:11.700542 4410 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.225.188" port: 36225 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:31:11.700920 4412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:11.700797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.188:36225 (local address 127.0.0.1:47824)
0928 06:31:11.700847 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:11.700891 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.188:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 06:31:11.701054 4358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.225.188:36225: connect: Connection refused (error 111)
I20260928 06:31:11.701535 4412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:11.701454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.188:36225 (local address 127.0.0.1:47826)
0928 06:31:11.701487 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:11.701512 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.188:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260928 06:31:11.701648 4410 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.225.188:36225: Network error: Client connection negotiation failed: client connection to 127.22.225.188:36225: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.188.keytab.
Entry for principal kudu/127.22.225.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.188.keytab.
I20260928 06:31:11.707424 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.188:36225
--webserver_interface=127.22.225.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.188.keytab
--principal=kudu/127.22.225.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:44143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.225.190:36075,127.22.225.189:33341,127.22.225.188:36225
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:31:11.712085 4335 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.225.189" port: 33341 } attempt: 3
Sep 28 06:31:11 dist-test-slave-d59m krb5kdc[4254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577071, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.190@KRBTEST.COM for kudu/127.22.225.189@KRBTEST.COM
I20260928 06:31:11.721172 4414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:11.712537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34332 (local address 127.22.225.189:33341)
0928 06:31:11.712633 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:11.712635 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:11.712648 (+ 13us) server_negotiation.cc:407] Connection header received
0928 06:31:11.712680 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:11.712681 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:11.712694 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:11.712768 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:11.713223 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:11.714055 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:11.715206 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:11.715571 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:11.715649 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:11.718144 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:11.718147 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:11.718147 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:11.718156 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:11.719874 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:11.720298 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:11.720299 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:11.720299 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:11.720359 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:11.720602 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:11.720603 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:11.720603 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:11.720742 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:11.720809 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:11.720990 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 06:31:11.721070 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 06:31:11.721210 4339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:11.712465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.189:33341 (local address 127.0.0.1:34332)
0928 06:31:11.712528 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:11.712539 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:11.712613 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:11.712799 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:11.712800 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:11.713017 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:11.713108 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:11.713111 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:11.714161 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:11.714163 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:11.715099 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:11.715103 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:11.715665 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:11.715667 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:11.715764 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:11.716305 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:11.716312 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:11.718038 (+ 1726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:11.719969 (+ 1931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:11.719971 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:11.719971 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:11.720218 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:11.720435 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:11.720436 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:11.720437 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:11.720510 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:11.720818 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:11.720820 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:11.720878 (+ 58us) client_negotiation.cc:769] Sending connection context
0928 06:31:11.720960 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 06:31:11.721087 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 06:31:11.721980 4335 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.225.188" port: 36225 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:31:11.722379 4339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:11.722263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.188:36225 (local address 127.0.0.1:47840)
0928 06:31:11.722321 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:11.722354 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.188:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 06:31:11.722723 4339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:11.722632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.188:36225 (local address 127.0.0.1:47850)
0928 06:31:11.722680 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:11.722705 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.188:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 06:31:11.722847 4335 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.225.188:36225: Network error: Client connection negotiation failed: client connection to 127.22.225.188:36225: connect: Connection refused (error 111)
I20260928 06:31:11.750909 4410 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.225.188" port: 36225 } attempt: 1
I20260928 06:31:11.751464 4412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:11.751314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.188:36225 (local address 127.0.0.1:47864)
0928 06:31:11.751372 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:11.751421 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.188:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 06:31:11.751814 4412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:11.751702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.188:36225 (local address 127.0.0.1:47874)
0928 06:31:11.751754 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:11.751792 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.188:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260928 06:31:11.751911 4410 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.225.188:36225: Network error: Client connection negotiation failed: client connection to 127.22.225.188:36225: connect: Connection refused (error 111)
I20260928 06:31:11.754011 4335 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.225.188" port: 36225 } attempt: 1
I20260928 06:31:11.754422 4339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:11.754276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.188:36225 (local address 127.0.0.1:47886)
0928 06:31:11.754348 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:11.754388 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.188:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 06:31:11.754767 4339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:11.754664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.188:36225 (local address 127.0.0.1:47902)
0928 06:31:11.754716 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:11.754745 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.188:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 06:31:11.754858 4335 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.225.188:36225: Network error: Client connection negotiation failed: client connection to 127.22.225.188:36225: connect: Connection refused (error 111)
W20260928 06:31:11.796166 4417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:11.796420 4417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:11.796464 4417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:11.798529 4417 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:31:11.798607 4417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:11.798642 4417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:11.798666 4417 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:31:11.798686 4417 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:31:11.800760 4417 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:44143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.22.225.190:36075,127.22.225.189:33341,127.22.225.188:36225
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.188.keytab
--principal=kudu/127.22.225.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.188:36225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.22.225.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:31:11.801028 4417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:11.801275 4417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:11.806087 4423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:11.806087 4425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:11.806154 4422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:11.806350 4417 server_base.cc:1034] running on GCE node
I20260928 06:31:11.806736 4417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:11.807041 4417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:11.808211 4417 hybrid_clock.cc:648] HybridClock initialized: now 1790577071808190 us; error 41 us; skew 500 ppm
Sep 28 06:31:11 dist-test-slave-d59m krb5kdc[4254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577071, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:11.810937 4417 init.cc:377] Logged in from keytab as kudu/127.22.225.188@KRBTEST.COM (short username kudu)
I20260928 06:31:11.811446 4417 webserver.cc:460] Webserver started at http://127.22.225.188:45383/ using document root <none> and password file <none>
I20260928 06:31:11.811682 4417 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:11.811754 4417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:11.811916 4417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:11.813187 4417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/minicluster-data/master-2/data/instance:
uuid: "b840f07be1354424a0287c2fe68153fc"
format_stamp: "Formatted at 2026-09-28 06:31:11 on dist-test-slave-d59m"
server_key: "e0d53fd23ebbdd5b63cee77e59adb32c"
server_key_iv: "a800d96970cf2c44c1eb68271e114d09"
server_key_version: "encryptionkey@0"
I20260928 06:31:11.813575 4417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/minicluster-data/master-2/wal/instance:
uuid: "b840f07be1354424a0287c2fe68153fc"
format_stamp: "Formatted at 2026-09-28 06:31:11 on dist-test-slave-d59m"
server_key: "e0d53fd23ebbdd5b63cee77e59adb32c"
server_key_iv: "a800d96970cf2c44c1eb68271e114d09"
server_key_version: "encryptionkey@0"
I20260928 06:31:11.814962 4417 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:31:11.815778 4432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:11.815948 4417 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:11.816056 4417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/minicluster-data/master-2/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/minicluster-data/master-2/wal
uuid: "b840f07be1354424a0287c2fe68153fc"
format_stamp: "Formatted at 2026-09-28 06:31:11 on dist-test-slave-d59m"
server_key: "e0d53fd23ebbdd5b63cee77e59adb32c"
server_key_iv: "a800d96970cf2c44c1eb68271e114d09"
server_key_version: "encryptionkey@0"
I20260928 06:31:11.816143 4417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:11.820070 4410 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.225.188" port: 36225 } attempt: 2
I20260928 06:31:11.820478 4412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:11.820333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.188:36225 (local address 127.0.0.1:47910)
0928 06:31:11.820398 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:11.820442 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.188:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 06:31:11.820833 4412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:11.820753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.188:36225 (local address 127.0.0.1:47920)
0928 06:31:11.820790 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:11.820811 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.188:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260928 06:31:11.820943 4410 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.225.188:36225: Network error: Client connection negotiation failed: client connection to 127.22.225.188:36225: connect: Connection refused (error 111)
I20260928 06:31:11.830024 4335 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.225.188" port: 36225 } attempt: 2
I20260928 06:31:11.830471 4339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:11.830342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.188:36225 (local address 127.0.0.1:47924)
0928 06:31:11.830404 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:11.830437 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.188:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 06:31:11.830922 4339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:11.830829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.188:36225 (local address 127.0.0.1:47938)
0928 06:31:11.830874 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:11.830898 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.188:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 06:31:11.830996 4417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
W20260928 06:31:11.831111 4335 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.225.188:36225: Network error: Client connection negotiation failed: client connection to 127.22.225.188:36225: connect: Connection refused (error 111)
I20260928 06:31:11.833869 4417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:11.834041 4417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:11.839708 4417 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.188:36225
I20260928 06:31:11.839748 4484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.188:36225 every 8 connection(s)
I20260928 06:31:11.840196 4417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/minicluster-data/master-2/data/info.pb
I20260928 06:31:11.841159 4485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:11.842758 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 4417
I20260928 06:31:11.842869 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576899143097-23430-0/minicluster-data/master-2/wal/instance
I20260928 06:31:11.843073 23430 external_mini_cluster.cc:1428] Setting key caff15f81491f77149e4cd5473879906
I20260928 06:31:11.842872 4485 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36075 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:31:11 dist-test-slave-d59m krb5kdc[4254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577071, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.188@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
Sep 28 06:31:11 dist-test-slave-d59m krb5kdc[4254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.225.188@KRBTEST.COM
I20260928 06:31:11.854357 4338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:11.843934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39700 (local address 127.22.225.190:36075)
0928 06:31:11.843983 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:11.843985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:11.844082 (+ 97us) server_negotiation.cc:407] Connection header received
0928 06:31:11.844763 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:11.844764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:11.844776 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:11.844825 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:11.845594 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:11.846359 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:11.847788 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:11.848075 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:11.848151 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:11.850533 (+ 2382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:11.850536 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:11.850536 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:11.850553 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:11.852641 (+ 2088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:11.853265 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:11.853266 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:11.853266 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:11.853324 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:11.853660 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:11.853662 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:11.853662 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:11.853819 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:11.853883 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:11.854173 (+ 290us) server_negotiation.cc:299] Negotiation successful
0928 06:31:11.854261 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 06:31:11.854580 4488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:11.843728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36075 (local address 127.0.0.1:39700)
0928 06:31:11.843933 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:11.843987 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:11.844647 (+ 660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:11.844858 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:11.844861 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:11.845272 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:11.845504 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:11.845511 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:11.846457 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:11.846458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:11.847689 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:11.847692 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:11.848172 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:11.848174 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:11.848253 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:11.848814 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:11.848822 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:11.850419 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:11.852730 (+ 2311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:11.852738 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:11.852741 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:11.853131 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:11.853408 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:11.853409 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:11.853409 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:11.853574 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:11.853874 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:11.853881 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:11.854033 (+ 152us) client_negotiation.cc:769] Sending connection context
0928 06:31:11.854149 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 06:31:11.854309 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":3,"thread_start_us":93,"threads_started":1}
I20260928 06:31:11.854933 4489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:11.843928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47950 (local address 127.22.225.188:36225)
0928 06:31:11.844060 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:11.844064 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:11.844096 (+ 32us) server_negotiation.cc:407] Connection header received
0928 06:31:11.844648 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:11.844662 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:11.844687 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:11.844795 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:11.845519 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:11.846666 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:11.847383 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:11.847833 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:11.847936 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:11.850821 (+ 2885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:11.850828 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:11.850828 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:11.850835 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:11.853103 (+ 2268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:11.853531 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:11.853534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:11.853535 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:11.853600 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:11.853819 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:11.853820 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:11.853820 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:11.854070 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:11.854226 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:11.854677 (+ 451us) server_negotiation.cc:299] Negotiation successful
0928 06:31:11.854762 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":104,"thread_start_us":65,"threads_started":1}
I20260928 06:31:11.855321 4485 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.225.189" port: 33341 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:31:11 dist-test-slave-d59m krb5kdc[4254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
Sep 28 06:31:11 dist-test-slave-d59m krb5kdc[4254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577071, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.188@KRBTEST.COM for kudu/127.22.225.189@KRBTEST.COM
I20260928 06:31:11.864725 4338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:11.856124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39716 (local address 127.22.225.190:36075)
0928 06:31:11.856184 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:11.856186 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:11.856194 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:31:11.856214 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:11.856214 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:11.856223 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:11.856279 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:11.856828 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:11.857306 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:11.858021 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:11.858221 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:11.858294 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:11.860511 (+ 2217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:11.860512 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:11.860513 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:11.860520 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:11.863468 (+ 2948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:11.863882 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:11.863883 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:11.863883 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:11.863943 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:11.864145 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:11.864146 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:11.864146 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:11.864259 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:11.864449 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:11.864500 (+ 51us) server_negotiation.cc:299] Negotiation successful
0928 06:31:11.864606 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 06:31:11.867889 4488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:11.855616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.189:33341 (local address 127.0.0.1:34340)
0928 06:31:11.855658 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:11.855668 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:11.855704 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:11.860349 (+ 4645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:11.860351 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:11.860508 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:11.860572 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:11.860593 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:11.861606 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:11.861606 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:11.862306 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:11.862326 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:11.862690 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:11.862691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:11.862766 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:11.863138 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:11.863144 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:11.864826 (+ 1682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:11.866819 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:11.866821 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:11.866821 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:11.867023 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:11.867242 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:11.867243 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:11.867243 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:11.867310 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:11.867561 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:11.867562 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:11.867604 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 06:31:11.867690 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 06:31:11.867788 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 06:31:11.867884 4414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:11.855704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34340 (local address 127.22.225.189:33341)
0928 06:31:11.860201 (+ 4497us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:11.860203 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:11.860216 (+ 13us) server_negotiation.cc:407] Connection header received
0928 06:31:11.860243 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:11.860245 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:11.860258 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:11.860327 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:11.860717 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:11.861511 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:11.862403 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:11.862606 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:11.862690 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:11.864942 (+ 2252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:11.864944 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:11.864945 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:11.864954 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:11.866729 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:11.867103 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:11.867104 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:11.867105 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:11.867167 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:11.867364 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:11.867365 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:11.867365 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:11.867479 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:11.867550 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:11.867717 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 06:31:11.867791 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4461}
I20260928 06:31:11.868701 4485 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.225.188" port: 36225 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:31:11 dist-test-slave-d59m krb5kdc[4254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.225.189@KRBTEST.COM
Sep 28 06:31:11 dist-test-slave-d59m krb5kdc[4254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577071, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.188@KRBTEST.COM for kudu/127.22.225.188@KRBTEST.COM
I20260928 06:31:11.874280 4490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:11.865378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34348 (local address 127.22.225.189:33341)
0928 06:31:11.865565 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:11.865566 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:11.865574 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:31:11.865595 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:11.865596 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:11.865608 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:11.865699 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:11.866240 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:11.867197 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:11.868034 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:11.868299 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:11.868368 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:11.870875 (+ 2507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:11.870876 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:11.870877 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:11.870885 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:11.872755 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:11.873105 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:11.873106 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:11.873106 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:11.873164 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:11.873365 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:11.873366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:11.873366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:11.873481 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:11.873659 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:11.873836 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 06:31:11.873916 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":113,"threads_started":1}
I20260928 06:31:11.875872 23430 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:31:11.876592 4488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:11.868988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.188:36225 (local address 127.0.0.1:47952)
0928 06:31:11.869059 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:11.869069 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:11.869113 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:11.869234 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:11.869235 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:11.869397 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:11.869424 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:11.869426 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:11.870334 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:11.870335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:11.870959 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:11.870962 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:11.872023 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:11.872025 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:11.872074 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:11.872438 (+ 364us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:11.872443 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:11.873766 (+ 1323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:11.875399 (+ 1633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:11.875399 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:11.875400 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:11.875576 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:11.875847 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:11.875847 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:11.875848 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:11.875934 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:11.876213 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:11.876214 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:11.876257 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 06:31:11.876330 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 06:31:11.876443 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260928 06:31:11.876662 4489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:11.869023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47952 (local address 127.22.225.188:36225)
0928 06:31:11.869063 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:11.869065 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:11.869098 (+ 33us) server_negotiation.cc:407] Connection header received
0928 06:31:11.869168 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:11.869168 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:11.869177 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:11.869230 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:11.869516 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:11.870251 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:11.871048 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:11.871655 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:11.872010 (+ 355us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:11.873853 (+ 1843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:11.873854 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:11.873854 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:11.873861 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:11.875316 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:11.875658 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:11.875659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:11.875660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:11.875724 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:11.876006 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:11.876007 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:11.876008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:11.876136 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:11.876207 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:11.876369 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 06:31:11.876505 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":25}
I20260928 06:31:11.878922 4485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b840f07be1354424a0287c2fe68153fc: Bootstrap starting.
I20260928 06:31:11.879974 4485 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b840f07be1354424a0287c2fe68153fc: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:11.880290 4485 log.cc:824] T 00000000000000000000000000000000 P b840f07be1354424a0287c2fe68153fc: Log is configured to *not* fsync() on all Append() calls
I20260928 06:31:11.881232 4485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b840f07be1354424a0287c2fe68153fc: No bootstrap required, opened a new log
I20260928 06:31:11.883867 4485 raft_consensus.cc:348] T 00000000000000000000000000000000 P b840f07be1354424a0287c2fe68153fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bd1e649645a4d5ca66a6e9a7dea54b3" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36075 } } peers { permanent_uuid: "6c16feb28f634d35adeb6b39e97b2409" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 33341 } } peers { permanent_uuid: "b840f07be1354424a0287c2fe68153fc" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 36225 } }
I20260928 06:31:11.884188 4485 raft_consensus.cc:374] T 00000000000000000000000000000000 P b840f07be1354424a0287c2fe68153fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:11.884289 4485 raft_consensus.cc:729] T 00000000000000000000000000000000 P b840f07be1354424a0287c2fe68153fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b840f07be1354424a0287c2fe68153fc, State: Initialized, Role: FOLLOWER
I20260928 06:31:11.884512 4485 consensus_queue.cc:260] T 00000000000000000000000000000000 P b840f07be1354424a0287c2fe68153fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bd1e649645a4d5ca66a6e9a7dea54b3" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36075 } } peers { permanent_uuid: "6c16feb28f634d35adeb6b39e97b2409" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 33341 } } peers { permanent_uuid: "b840f07be1354424a0287c2fe68153fc" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 36225 } }
I20260928 06:31:11.884959 4499 sys_catalog.cc:455] T 00000000000000000000000000000000 P b840f07be1354424a0287c2fe68153fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bd1e649645a4d5ca66a6e9a7dea54b3" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36075 } } peers { permanent_uuid: "6c16feb28f634d35adeb6b39e97b2409" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 33341 } } peers { permanent_uuid: "b840f07be1354424a0287c2fe68153fc" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 36225 } } }
I20260928 06:31:11.885030 4485 sys_catalog.cc:564] T 00000000000000000000000000000000 P b840f07be1354424a0287c2fe68153fc [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:31:11.885061 4499 sys_catalog.cc:458] T 00000000000000000000000000000000 P b840f07be1354424a0287c2fe68153fc [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:31:11.885360 4489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:11.877693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47958 (local address 127.22.225.188:36225)
0928 06:31:11.877750 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:11.877752 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:11.877759 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:31:11.877779 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:11.877780 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:11.877789 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:11.877843 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:11.878536 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:11.879128 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:11.880027 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:11.880277 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:11.880320 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:11.882504 (+ 2184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:11.882505 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:11.882506 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:11.882512 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:11.884057 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:11.884423 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:11.884424 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:11.884425 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:11.884485 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:11.884744 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:11.884744 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:11.884745 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:11.884900 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:11.884979 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:11.885200 (+ 221us) server_negotiation.cc:299] Negotiation successful
0928 06:31:11.885285 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 06:31:11.886340 4338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:11.877706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39722 (local address 127.22.225.190:36075)
0928 06:31:11.877896 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:11.877898 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:11.877907 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:31:11.877932 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:11.877933 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:11.877944 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:11.878000 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:11.878659 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:11.879307 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:11.880314 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:11.880955 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:11.881025 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:11.882468 (+ 1443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:11.882470 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:11.882470 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:11.882479 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:11.884634 (+ 2155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:11.885025 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:11.885026 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:11.885027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:11.885086 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:11.885301 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:11.885303 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:11.885303 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:11.885433 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:11.885533 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:11.886135 (+ 602us) server_negotiation.cc:299] Negotiation successful
0928 06:31:11.886231 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163}
W20260928 06:31:11.887459 4510 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b840f07be1354424a0287c2fe68153fc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:31:11.887517 4510 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:31:11.888468 4417 master_runner.cc:186] Error getting master registration for 127.22.225.190:36075: OK, instance_id { permanent_uuid: "1bd1e649645a4d5ca66a6e9a7dea54b3" instance_seqno: 1790577071483431 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 06:31:11.890710 4417 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 06:31:11.893391 4490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:11.878177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34360 (local address 127.22.225.189:33341)
0928 06:31:11.881794 (+ 3617us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:11.881796 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:11.881902 (+ 106us) server_negotiation.cc:407] Connection header received
0928 06:31:11.881995 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:11.881996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:11.882007 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:11.882039 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:11.887166 (+ 5127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:11.887773 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:11.888472 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:11.888823 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:11.888888 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:11.890411 (+ 1523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:11.890413 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:11.890413 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:11.890420 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:11.892120 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:11.892507 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:11.892508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:11.892508 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:11.892597 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:11.892827 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:11.892828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:11.892829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:11.892965 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:11.893079 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:11.893231 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 06:31:11.893312 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3592}
I20260928 06:31:11.912108 4410 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.225.188" port: 36225 } attempt: 3
Sep 28 06:31:11 dist-test-slave-d59m krb5kdc[4254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577071, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.189@KRBTEST.COM for kudu/127.22.225.188@KRBTEST.COM
I20260928 06:31:11.920802 4412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:11.912405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.188:36225 (local address 127.0.0.1:47968)
0928 06:31:11.912460 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:11.912471 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:11.912509 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:11.912700 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:11.912701 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:11.912897 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:11.912954 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:11.912956 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:11.913906 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:11.913907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:11.914725 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:11.914728 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:11.915253 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:11.915255 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:11.915318 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:11.915839 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:11.915845 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:11.917504 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:11.919539 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:11.919541 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:11.919541 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:11.919780 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:11.920058 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:11.920058 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:11.920059 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:11.920133 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:11.920438 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:11.920438 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:11.920490 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 06:31:11.920563 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 06:31:11.920695 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 06:31:11.920887 4489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:11.912500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47968 (local address 127.22.225.188:36225)
0928 06:31:11.912556 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:11.912558 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:11.912567 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:31:11.912608 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:11.912608 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:11.912619 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:11.912683 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:11.913053 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:11.913817 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:11.914821 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:11.915155 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:11.915230 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:11.917608 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:11.917609 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:11.917610 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:11.917618 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:11.919449 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:11.919905 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:11.919906 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:11.919907 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:11.919974 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:11.920208 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:11.920208 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:11.920209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:11.920365 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:11.920433 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:11.920649 (+ 216us) server_negotiation.cc:299] Negotiation successful
0928 06:31:11.920743 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 06:31:11.923586 4410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c16feb28f634d35adeb6b39e97b2409: Bootstrap starting.
I20260928 06:31:11.924479 4410 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c16feb28f634d35adeb6b39e97b2409: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:11.924877 4410 log.cc:824] T 00000000000000000000000000000000 P 6c16feb28f634d35adeb6b39e97b2409: Log is configured to *not* fsync() on all Append() calls
I20260928 06:31:11.925884 4410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c16feb28f634d35adeb6b39e97b2409: No bootstrap required, opened a new log
I20260928 06:31:11.928318 4410 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c16feb28f634d35adeb6b39e97b2409 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bd1e649645a4d5ca66a6e9a7dea54b3" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36075 } } peers { permanent_uuid: "6c16feb28f634d35adeb6b39e97b2409" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 33341 } } peers { permanent_uuid: "b840f07be1354424a0287c2fe68153fc" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 36225 } }
I20260928 06:31:11.928562 4410 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c16feb28f634d35adeb6b39e97b2409 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:11.928694 4410 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c16feb28f634d35adeb6b39e97b2409 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c16feb28f634d35adeb6b39e97b2409, State: Initialized, Role: FOLLOWER
I20260928 06:31:11.928880 4410 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c16feb28f634d35adeb6b39e97b2409 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bd1e649645a4d5ca66a6e9a7dea54b3" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36075 } } peers { permanent_uuid: "6c16feb28f634d35adeb6b39e97b2409" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 33341 } } peers { permanent_uuid: "b840f07be1354424a0287c2fe68153fc" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 36225 } }
I20260928 06:31:11.929231 4513 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c16feb28f634d35adeb6b39e97b2409 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bd1e649645a4d5ca66a6e9a7dea54b3" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36075 } } peers { permanent_uuid: "6c16feb28f634d35adeb6b39e97b2409" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 33341 } } peers { permanent_uuid: "b840f07be1354424a0287c2fe68153fc" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 36225 } } }
I20260928 06:31:11.929333 4513 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c16feb28f634d35adeb6b39e97b2409 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:31:11.929409 4410 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c16feb28f634d35adeb6b39e97b2409 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:31:11.930250 4335 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.225.188" port: 36225 } attempt: 3
W20260928 06:31:11.931658 4524 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6c16feb28f634d35adeb6b39e97b2409: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:31:11.931725 4524 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:31:11.931820 4342 master_runner.cc:186] Error getting master registration for 127.22.225.190:36075: OK, instance_id { permanent_uuid: "1bd1e649645a4d5ca66a6e9a7dea54b3" instance_seqno: 1790577071483431 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 06:31:11.934125 4342 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 28 06:31:11 dist-test-slave-d59m krb5kdc[4254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577071, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.190@KRBTEST.COM for kudu/127.22.225.188@KRBTEST.COM
I20260928 06:31:11.939723 4339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:11.930563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.188:36225 (local address 127.0.0.1:47978)
0928 06:31:11.930672 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:11.930684 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:11.930732 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:11.931109 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:11.931110 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:11.931310 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:11.931361 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:11.931363 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:11.932336 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:11.932338 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:11.933223 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:11.933225 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:11.933707 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:11.933708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:11.933764 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:11.934339 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:11.934344 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:11.935966 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:11.938567 (+ 2601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:11.938568 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:11.938568 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:11.938798 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:11.939085 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:11.939086 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:11.939086 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:11.939149 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:11.939429 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:11.939430 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:11.939473 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 06:31:11.939520 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 06:31:11.939620 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260928 06:31:11.940413 4489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:11.930836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47978 (local address 127.22.225.188:36225)
0928 06:31:11.930928 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:11.930930 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:11.930939 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:31:11.930966 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:11.930967 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:11.930977 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:11.931097 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:11.931449 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:11.932228 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:11.933323 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:11.933623 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:11.933932 (+ 309us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:11.936073 (+ 2141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:11.936075 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:11.936076 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:11.936083 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:11.938466 (+ 2383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:11.938882 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:11.938883 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:11.938883 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:11.938996 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:11.939222 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:11.939223 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:11.939223 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:11.939355 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:11.940202 (+ 847us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:11.940246 (+ 44us) server_negotiation.cc:299] Negotiation successful
0928 06:31:11.940323 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 06:31:11.941896 4335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bd1e649645a4d5ca66a6e9a7dea54b3: Bootstrap starting.
I20260928 06:31:11.942562 4335 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1bd1e649645a4d5ca66a6e9a7dea54b3: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:11.942878 4335 log.cc:824] T 00000000000000000000000000000000 P 1bd1e649645a4d5ca66a6e9a7dea54b3: Log is configured to *not* fsync() on all Append() calls
I20260928 06:31:11.943825 4335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bd1e649645a4d5ca66a6e9a7dea54b3: No bootstrap required, opened a new log
I20260928 06:31:11.945966 4335 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1bd1e649645a4d5ca66a6e9a7dea54b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bd1e649645a4d5ca66a6e9a7dea54b3" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36075 } } peers { permanent_uuid: "6c16feb28f634d35adeb6b39e97b2409" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 33341 } } peers { permanent_uuid: "b840f07be1354424a0287c2fe68153fc" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 36225 } }
I20260928 06:31:11.946128 4335 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1bd1e649645a4d5ca66a6e9a7dea54b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:11.946173 4335 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1bd1e649645a4d5ca66a6e9a7dea54b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bd1e649645a4d5ca66a6e9a7dea54b3, State: Initialized, Role: FOLLOWER
I20260928 06:31:11.946309 4335 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1bd1e649645a4d5ca66a6e9a7dea54b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bd1e649645a4d5ca66a6e9a7dea54b3" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36075 } } peers { permanent_uuid: "6c16feb28f634d35adeb6b39e97b2409" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 33341 } } peers { permanent_uuid: "b840f07be1354424a0287c2fe68153fc" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 36225 } }
I20260928 06:31:11.946638 4335 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1bd1e649645a4d5ca66a6e9a7dea54b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:31:11.946607 4527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bd1e649645a4d5ca66a6e9a7dea54b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bd1e649645a4d5ca66a6e9a7dea54b3" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36075 } } peers { permanent_uuid: "6c16feb28f634d35adeb6b39e97b2409" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 33341 } } peers { permanent_uuid: "b840f07be1354424a0287c2fe68153fc" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 36225 } } }
I20260928 06:31:11.946791 4527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bd1e649645a4d5ca66a6e9a7dea54b3 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:31:11.953523 4267 master_runner.cc:263] No leader master found from master 1bd1e649645a4d5ca66a6e9a7dea54b3
I20260928 06:31:11.953586 4267 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 06:31:11.957221 4538 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1bd1e649645a4d5ca66a6e9a7dea54b3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:31:11.957288 4538 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260928 06:31:11.980908 4288 tablet.cc:2369] T 00000000000000000000000000000000 P 1bd1e649645a4d5ca66a6e9a7dea54b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:31:12.084836 4438 tablet.cc:2369] T 00000000000000000000000000000000 P b840f07be1354424a0287c2fe68153fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:31:12.171645 4363 tablet.cc:2369] T 00000000000000000000000000000000 P 6c16feb28f634d35adeb6b39e97b2409: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:31:12.234751 4499 raft_consensus.cc:482] T 00000000000000000000000000000000 P b840f07be1354424a0287c2fe68153fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:31:12.234902 4499 raft_consensus.cc:504] T 00000000000000000000000000000000 P b840f07be1354424a0287c2fe68153fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bd1e649645a4d5ca66a6e9a7dea54b3" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36075 } } peers { permanent_uuid: "6c16feb28f634d35adeb6b39e97b2409" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 33341 } } peers { permanent_uuid: "b840f07be1354424a0287c2fe68153fc" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 36225 } }
I20260928 06:31:12.235175 4499 leader_election.cc:288] T 00000000000000000000000000000000 P b840f07be1354424a0287c2fe68153fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1bd1e649645a4d5ca66a6e9a7dea54b3 (127.22.225.190:36075), 6c16feb28f634d35adeb6b39e97b2409 (127.22.225.189:33341)
I20260928 06:31:12.235514 4384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b840f07be1354424a0287c2fe68153fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c16feb28f634d35adeb6b39e97b2409" is_pre_election: true
I20260928 06:31:12.235670 4384 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6c16feb28f634d35adeb6b39e97b2409 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b840f07be1354424a0287c2fe68153fc in term 0.
I20260928 06:31:12.235848 4435 leader_election.cc:302] T 00000000000000000000000000000000 P b840f07be1354424a0287c2fe68153fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c16feb28f634d35adeb6b39e97b2409, b840f07be1354424a0287c2fe68153fc; no voters:
I20260928 06:31:12.236074 4499 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b840f07be1354424a0287c2fe68153fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:31:12.236130 4499 raft_consensus.cc:482] T 00000000000000000000000000000000 P b840f07be1354424a0287c2fe68153fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:31:12.236151 4499 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b840f07be1354424a0287c2fe68153fc [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:12.236645 4309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b840f07be1354424a0287c2fe68153fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bd1e649645a4d5ca66a6e9a7dea54b3" is_pre_election: true
I20260928 06:31:12.236789 4309 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1bd1e649645a4d5ca66a6e9a7dea54b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b840f07be1354424a0287c2fe68153fc in term 0.
I20260928 06:31:12.237236 4499 raft_consensus.cc:504] T 00000000000000000000000000000000 P b840f07be1354424a0287c2fe68153fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bd1e649645a4d5ca66a6e9a7dea54b3" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36075 } } peers { permanent_uuid: "6c16feb28f634d35adeb6b39e97b2409" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 33341 } } peers { permanent_uuid: "b840f07be1354424a0287c2fe68153fc" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 36225 } }
I20260928 06:31:12.237377 4499 leader_election.cc:288] T 00000000000000000000000000000000 P b840f07be1354424a0287c2fe68153fc [CANDIDATE]: Term 1 election: Requested vote from peers 1bd1e649645a4d5ca66a6e9a7dea54b3 (127.22.225.190:36075), 6c16feb28f634d35adeb6b39e97b2409 (127.22.225.189:33341)
I20260928 06:31:12.237510 4309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b840f07be1354424a0287c2fe68153fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bd1e649645a4d5ca66a6e9a7dea54b3"
I20260928 06:31:12.237596 4309 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1bd1e649645a4d5ca66a6e9a7dea54b3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:12.237746 4384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b840f07be1354424a0287c2fe68153fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c16feb28f634d35adeb6b39e97b2409"
I20260928 06:31:12.237830 4384 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c16feb28f634d35adeb6b39e97b2409 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:12.238503 4309 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1bd1e649645a4d5ca66a6e9a7dea54b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b840f07be1354424a0287c2fe68153fc in term 1.
I20260928 06:31:12.238734 4433 leader_election.cc:302] T 00000000000000000000000000000000 P b840f07be1354424a0287c2fe68153fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bd1e649645a4d5ca66a6e9a7dea54b3, b840f07be1354424a0287c2fe68153fc; no voters:
I20260928 06:31:12.238806 4384 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6c16feb28f634d35adeb6b39e97b2409 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b840f07be1354424a0287c2fe68153fc in term 1.
I20260928 06:31:12.239076 4499 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b840f07be1354424a0287c2fe68153fc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:31:12.239212 4499 raft_consensus.cc:686] T 00000000000000000000000000000000 P b840f07be1354424a0287c2fe68153fc [term 1 LEADER]: Becoming Leader. State: Replica: b840f07be1354424a0287c2fe68153fc, State: Running, Role: LEADER
I20260928 06:31:12.239317 4499 consensus_queue.cc:237] T 00000000000000000000000000000000 P b840f07be1354424a0287c2fe68153fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bd1e649645a4d5ca66a6e9a7dea54b3" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36075 } } peers { permanent_uuid: "6c16feb28f634d35adeb6b39e97b2409" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 33341 } } peers { permanent_uuid: "b840f07be1354424a0287c2fe68153fc" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 36225 } }
I20260928 06:31:12.239778 4540 sys_catalog.cc:455] T 00000000000000000000000000000000 P b840f07be1354424a0287c2fe68153fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader b840f07be1354424a0287c2fe68153fc. Latest consensus state: current_term: 1 leader_uuid: "b840f07be1354424a0287c2fe68153fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bd1e649645a4d5ca66a6e9a7dea54b3" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36075 } } peers { permanent_uuid: "6c16feb28f634d35adeb6b39e97b2409" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 33341 } } peers { permanent_uuid: "b840f07be1354424a0287c2fe68153fc" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 36225 } } }
I20260928 06:31:12.239878 4540 sys_catalog.cc:458] T 00000000000000000000000000000000 P b840f07be1354424a0287c2fe68153fc [sys.catalog]: This master's current role is: LEADER
I20260928 06:31:12.240115 4542 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:31:12.240320 4542 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:31:12.241964 4309 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1bd1e649645a4d5ca66a6e9a7dea54b3 [term 1 FOLLOWER]: Refusing update from remote peer b840f07be1354424a0287c2fe68153fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:31:12.242074 4384 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6c16feb28f634d35adeb6b39e97b2409 [term 1 FOLLOWER]: Refusing update from remote peer b840f07be1354424a0287c2fe68153fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:31:12.242421 4540 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b840f07be1354424a0287c2fe68153fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c16feb28f634d35adeb6b39e97b2409" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 33341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:31:12.242580 4499 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b840f07be1354424a0287c2fe68153fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bd1e649645a4d5ca66a6e9a7dea54b3" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:31:12.243857 4527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bd1e649645a4d5ca66a6e9a7dea54b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b840f07be1354424a0287c2fe68153fc. Latest consensus state: current_term: 1 leader_uuid: "b840f07be1354424a0287c2fe68153fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bd1e649645a4d5ca66a6e9a7dea54b3" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36075 } } peers { permanent_uuid: "6c16feb28f634d35adeb6b39e97b2409" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 33341 } } peers { permanent_uuid: "b840f07be1354424a0287c2fe68153fc" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 36225 } } }
I20260928 06:31:12.243860 4513 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c16feb28f634d35adeb6b39e97b2409 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b840f07be1354424a0287c2fe68153fc. Latest consensus state: current_term: 1 leader_uuid: "b840f07be1354424a0287c2fe68153fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bd1e649645a4d5ca66a6e9a7dea54b3" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36075 } } peers { permanent_uuid: "6c16feb28f634d35adeb6b39e97b2409" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 33341 } } peers { permanent_uuid: "b840f07be1354424a0287c2fe68153fc" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 36225 } } }
I20260928 06:31:12.243960 4527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bd1e649645a4d5ca66a6e9a7dea54b3 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:31:12.243968 4513 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c16feb28f634d35adeb6b39e97b2409 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:31:12.244925 4543 mvcc.cc:205] Tried to move back new op lower bound from 7334203687900491776 to 7334203687892738048. Current Snapshot: MvccSnapshot[applied={T|T < 7334203687900491776}]
I20260928 06:31:12.245047 4499 sys_catalog.cc:455] T 00000000000000000000000000000000 P b840f07be1354424a0287c2fe68153fc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b840f07be1354424a0287c2fe68153fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bd1e649645a4d5ca66a6e9a7dea54b3" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36075 } } peers { permanent_uuid: "6c16feb28f634d35adeb6b39e97b2409" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 33341 } } peers { permanent_uuid: "b840f07be1354424a0287c2fe68153fc" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 36225 } } }
I20260928 06:31:12.245133 4499 sys_catalog.cc:458] T 00000000000000000000000000000000 P b840f07be1354424a0287c2fe68153fc [sys.catalog]: This master's current role is: LEADER
I20260928 06:31:12.245223 4540 sys_catalog.cc:455] T 00000000000000000000000000000000 P b840f07be1354424a0287c2fe68153fc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b840f07be1354424a0287c2fe68153fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bd1e649645a4d5ca66a6e9a7dea54b3" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36075 } } peers { permanent_uuid: "6c16feb28f634d35adeb6b39e97b2409" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 33341 } } peers { permanent_uuid: "b840f07be1354424a0287c2fe68153fc" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 36225 } } }
I20260928 06:31:12.245306 4540 sys_catalog.cc:458] T 00000000000000000000000000000000 P b840f07be1354424a0287c2fe68153fc [sys.catalog]: This master's current role is: LEADER
I20260928 06:31:12.245476 4542 catalog_manager.cc:1347] Generated new cluster ID: 80bfeae137e94e9f804c0c5020b2e1ec
I20260928 06:31:12.245579 4542 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:31:12.245762 4527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bd1e649645a4d5ca66a6e9a7dea54b3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b840f07be1354424a0287c2fe68153fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bd1e649645a4d5ca66a6e9a7dea54b3" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36075 } } peers { permanent_uuid: "6c16feb28f634d35adeb6b39e97b2409" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 33341 } } peers { permanent_uuid: "b840f07be1354424a0287c2fe68153fc" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 36225 } } }
I20260928 06:31:12.245854 4527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bd1e649645a4d5ca66a6e9a7dea54b3 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:31:12.245898 4545 mvcc.cc:205] Tried to move back new op lower bound from 7334203687900491776 to 7334203687892738048. Current Snapshot: MvccSnapshot[applied={T|T < 7334203687900491776}]
I20260928 06:31:12.245982 4513 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c16feb28f634d35adeb6b39e97b2409 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b840f07be1354424a0287c2fe68153fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bd1e649645a4d5ca66a6e9a7dea54b3" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36075 } } peers { permanent_uuid: "6c16feb28f634d35adeb6b39e97b2409" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 33341 } } peers { permanent_uuid: "b840f07be1354424a0287c2fe68153fc" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 36225 } } }
I20260928 06:31:12.246058 4513 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c16feb28f634d35adeb6b39e97b2409 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:31:12.268003 4542 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:31:12.268754 4542 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:31:12.276346 4542 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b840f07be1354424a0287c2fe68153fc: Generated new TSK 0
I20260928 06:31:12.276654 4542 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:31:12.932181 4524 catalog_manager.cc:1259] Loaded cluster ID: 80bfeae137e94e9f804c0c5020b2e1ec
I20260928 06:31:12.932284 4524 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6c16feb28f634d35adeb6b39e97b2409: loading cluster ID for follower catalog manager: success
I20260928 06:31:12.933229 4524 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6c16feb28f634d35adeb6b39e97b2409: acquiring CA information for follower catalog manager: success
I20260928 06:31:12.933681 4524 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6c16feb28f634d35adeb6b39e97b2409: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 06:31:12.957741 4538 catalog_manager.cc:1259] Loaded cluster ID: 80bfeae137e94e9f804c0c5020b2e1ec
I20260928 06:31:12.957842 4538 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1bd1e649645a4d5ca66a6e9a7dea54b3: loading cluster ID for follower catalog manager: success
I20260928 06:31:12.958652 4538 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1bd1e649645a4d5ca66a6e9a7dea54b3: acquiring CA information for follower catalog manager: success
I20260928 06:31:12.958997 4538 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1bd1e649645a4d5ca66a6e9a7dea54b3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 06:31:13.333168 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 4267
I20260928 06:31:13.339735 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 4342
I20260928 06:31:13.345921 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 4417
2026-09-28T06:31:13Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3182 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790576899143097-23430-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:31:13 dist-test-slave-d59m krb5kdc[4557](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:31:13 dist-test-slave-d59m krb5kdc[4557](info): set up 2 sockets
Sep 28 06:31:13 dist-test-slave-d59m krb5kdc[4557](info): commencing operation
krb5kdc: starting...
W20260928 06:31:14.417611 23430 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Sep 28 06:31:14 dist-test-slave-d59m krb5kdc[4557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:31:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:31:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab.
Entry for principal kudu/127.22.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab.
I20260928 06:31:14.555171 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790576899143097-23430-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790576899143097-23430-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790576899143097-23430-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:36967
--webserver_interface=127.22.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab
--principal=kudu/127.22.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:46317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.225.190:36967
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:31:14.645108 4570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:14.645358 4570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:14.645397 4570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:14.646605 4570 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:31:14.646672 4570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:14.646695 4570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:14.646723 4570 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:31:14.646755 4570 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:31:14.648265 4570 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:46317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790576899143097-23430-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790576899143097-23430-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.225.190:36967
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab
--principal=kudu/127.22.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:36967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790576899143097-23430-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:31:14.648559 4570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:14.648824 4570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:14.653707 4575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:14.653748 4576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:14.653771 4578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:14.654145 4570 server_base.cc:1034] running on GCE node
I20260928 06:31:14.654412 4570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:14.654757 4570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:14.655938 4570 hybrid_clock.cc:648] HybridClock initialized: now 1790577074655911 us; error 42 us; skew 500 ppm
Sep 28 06:31:14 dist-test-slave-d59m krb5kdc[4557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577074, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:14.658790 4570 init.cc:377] Logged in from keytab as kudu/127.22.225.190@KRBTEST.COM (short username kudu)
I20260928 06:31:14.659248 4570 webserver.cc:460] Webserver started at http://127.22.225.190:45927/ using document root <none> and password file <none>
I20260928 06:31:14.659493 4570 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:14.659564 4570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:14.659716 4570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:14.660883 4570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790576899143097-23430-0/minicluster-data/master-0/data/instance:
uuid: "82c499e644da45ad9524626665f7c5be"
format_stamp: "Formatted at 2026-09-28 06:31:14 on dist-test-slave-d59m"
server_key: "7f174a8a7d08915e34d4c9af9118e771"
server_key_iv: "a18df540c9b1da4a4ce3185452041ceb"
server_key_version: "encryptionkey@0"
I20260928 06:31:14.661254 4570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790576899143097-23430-0/minicluster-data/master-0/wal/instance:
uuid: "82c499e644da45ad9524626665f7c5be"
format_stamp: "Formatted at 2026-09-28 06:31:14 on dist-test-slave-d59m"
server_key: "7f174a8a7d08915e34d4c9af9118e771"
server_key_iv: "a18df540c9b1da4a4ce3185452041ceb"
server_key_version: "encryptionkey@0"
I20260928 06:31:14.662640 4570 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:31:14.663475 4585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:14.663666 4570 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:14.663775 4570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790576899143097-23430-0/minicluster-data/master-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790576899143097-23430-0/minicluster-data/master-0/wal
uuid: "82c499e644da45ad9524626665f7c5be"
format_stamp: "Formatted at 2026-09-28 06:31:14 on dist-test-slave-d59m"
server_key: "7f174a8a7d08915e34d4c9af9118e771"
server_key_iv: "a18df540c9b1da4a4ce3185452041ceb"
server_key_version: "encryptionkey@0"
I20260928 06:31:14.663851 4570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790576899143097-23430-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790576899143097-23430-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790576899143097-23430-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:14.680003 4570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:14.682363 4570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:14.682541 4570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:14.687431 4570 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.190:36967
I20260928 06:31:14.687467 4637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.190:36967 every 8 connection(s)
I20260928 06:31:14.687858 4570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
I20260928 06:31:14.688869 4638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:14.691198 4638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82c499e644da45ad9524626665f7c5be: Bootstrap starting.
I20260928 06:31:14.691514 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 4570
I20260928 06:31:14.691622 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790576899143097-23430-0/minicluster-data/master-0/wal/instance
I20260928 06:31:14.691820 23430 external_mini_cluster.cc:1428] Setting key 553d60a05722bb741efee385bb32cd5b
I20260928 06:31:14.691871 4638 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82c499e644da45ad9524626665f7c5be: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:14.692193 4638 log.cc:824] T 00000000000000000000000000000000 P 82c499e644da45ad9524626665f7c5be: Log is configured to *not* fsync() on all Append() calls
I20260928 06:31:14.693147 4638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82c499e644da45ad9524626665f7c5be: No bootstrap required, opened a new log
I20260928 06:31:14.695484 4638 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82c499e644da45ad9524626665f7c5be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82c499e644da45ad9524626665f7c5be" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36967 } }
I20260928 06:31:14.695652 4638 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82c499e644da45ad9524626665f7c5be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:14.695683 4638 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82c499e644da45ad9524626665f7c5be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82c499e644da45ad9524626665f7c5be, State: Initialized, Role: FOLLOWER
I20260928 06:31:14.695811 4638 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82c499e644da45ad9524626665f7c5be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82c499e644da45ad9524626665f7c5be" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36967 } }
I20260928 06:31:14.695902 4638 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82c499e644da45ad9524626665f7c5be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:31:14.695981 4638 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82c499e644da45ad9524626665f7c5be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:31:14.696036 4638 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82c499e644da45ad9524626665f7c5be [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:14.696815 4638 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82c499e644da45ad9524626665f7c5be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82c499e644da45ad9524626665f7c5be" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36967 } }
I20260928 06:31:14.696949 4638 leader_election.cc:302] T 00000000000000000000000000000000 P 82c499e644da45ad9524626665f7c5be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82c499e644da45ad9524626665f7c5be; no voters:
I20260928 06:31:14.697162 4638 leader_election.cc:288] T 00000000000000000000000000000000 P 82c499e644da45ad9524626665f7c5be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:31:14.697186 4643 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82c499e644da45ad9524626665f7c5be [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:31:14.697366 4643 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82c499e644da45ad9524626665f7c5be [term 1 LEADER]: Becoming Leader. State: Replica: 82c499e644da45ad9524626665f7c5be, State: Running, Role: LEADER
I20260928 06:31:14.697499 4643 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82c499e644da45ad9524626665f7c5be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82c499e644da45ad9524626665f7c5be" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36967 } }
I20260928 06:31:14.697526 4638 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82c499e644da45ad9524626665f7c5be [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:31:14.699363 4644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82c499e644da45ad9524626665f7c5be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82c499e644da45ad9524626665f7c5be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82c499e644da45ad9524626665f7c5be" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36967 } } }
I20260928 06:31:14.699455 4644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82c499e644da45ad9524626665f7c5be [sys.catalog]: This master's current role is: LEADER
I20260928 06:31:14.699590 4644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82c499e644da45ad9524626665f7c5be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82c499e644da45ad9524626665f7c5be. Latest consensus state: current_term: 1 leader_uuid: "82c499e644da45ad9524626665f7c5be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82c499e644da45ad9524626665f7c5be" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36967 } } }
I20260928 06:31:14.699653 4644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82c499e644da45ad9524626665f7c5be [sys.catalog]: This master's current role is: LEADER
Sep 28 06:31:14 dist-test-slave-d59m krb5kdc[4557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:31:14.700261 4657 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:31:14.700897 4657 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:31:14.702426 4657 catalog_manager.cc:1347] Generated new cluster ID: 2dca9ea228a942ffa56925422c72a1a0
I20260928 06:31:14.702478 4657 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:31:14.704197 4642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:14.692386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40796 (local address 127.22.225.190:36967)
0928 06:31:14.692660 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:14.692667 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:14.692694 (+ 27us) server_negotiation.cc:407] Connection header received
0928 06:31:14.693306 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:14.693315 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:14.693377 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:14.693529 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:14.694269 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:14.695735 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:14.696916 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:14.697524 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:14.697960 (+ 436us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:14.700389 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:14.700394 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:14.700394 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:14.700406 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:14.702350 (+ 1944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:14.702739 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:14.702742 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:14.702742 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:14.702842 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:14.703031 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:14.703041 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:14.703042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:14.703328 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:14.703449 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:14.703853 (+ 404us) server_negotiation.cc:299] Negotiation successful
0928 06:31:14.703982 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":145,"threads_started":1}
I20260928 06:31:14.705487 23430 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:31:14.713754 4657 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:31:14.714295 4657 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:31:14.718439 4657 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82c499e644da45ad9524626665f7c5be: Generated new TSK 0
I20260928 06:31:14.718636 4657 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:31:14.759680 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 4570
2026-09-28T06:31:14Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1409 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:31:14 dist-test-slave-d59m krb5kdc[4666](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:31:14 dist-test-slave-d59m krb5kdc[4666](info): set up 2 sockets
Sep 28 06:31:14 dist-test-slave-d59m krb5kdc[4666](info): commencing operation
krb5kdc: starting...
W20260928 06:31:15.841130 23430 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Sep 28 06:31:15 dist-test-slave-d59m krb5kdc[4666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:31:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:31:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab.
Entry for principal kudu/127.22.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab.
I20260928 06:31:15.966969 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:36479
--webserver_interface=127.22.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab
--principal=kudu/127.22.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:37031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.225.190:36479
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:31:16.051456 4679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:16.051728 4679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:16.051770 4679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:16.053056 4679 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:31:16.053119 4679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:16.053144 4679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:16.053175 4679 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:31:16.053205 4679 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:31:16.054745 4679 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:37031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.225.190:36479
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab
--principal=kudu/127.22.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:36479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:31:16.055027 4679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:16.055320 4679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:16.060343 4684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:16.060333 4687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:16.060554 4685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:16.060819 4679 server_base.cc:1034] running on GCE node
I20260928 06:31:16.061095 4679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:16.061421 4679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:16.062587 4679 hybrid_clock.cc:648] HybridClock initialized: now 1790577076062564 us; error 44 us; skew 500 ppm
Sep 28 06:31:16 dist-test-slave-d59m krb5kdc[4666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577076, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:16.069507 4679 init.cc:377] Logged in from keytab as kudu/127.22.225.190@KRBTEST.COM (short username kudu)
I20260928 06:31:16.070168 4679 webserver.cc:460] Webserver started at http://127.22.225.190:41111/ using document root <none> and password file <none>
I20260928 06:31:16.070461 4679 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:16.070536 4679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:16.070703 4679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:16.072330 4679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/master-0/data/instance:
uuid: "fc5064433d164e12b54242dc5cb206fd"
format_stamp: "Formatted at 2026-09-28 06:31:16 on dist-test-slave-d59m"
server_key: "8ff09d423c9b79596a1c63fe6d58dae1"
server_key_iv: "062ab7b58b2114e58aadc43f1f66401b"
server_key_version: "encryptionkey@0"
I20260928 06:31:16.072819 4679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/master-0/wal/instance:
uuid: "fc5064433d164e12b54242dc5cb206fd"
format_stamp: "Formatted at 2026-09-28 06:31:16 on dist-test-slave-d59m"
server_key: "8ff09d423c9b79596a1c63fe6d58dae1"
server_key_iv: "062ab7b58b2114e58aadc43f1f66401b"
server_key_version: "encryptionkey@0"
I20260928 06:31:16.074692 4679 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260928 06:31:16.075659 4694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:16.075842 4679 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:16.075949 4679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/master-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/master-0/wal
uuid: "fc5064433d164e12b54242dc5cb206fd"
format_stamp: "Formatted at 2026-09-28 06:31:16 on dist-test-slave-d59m"
server_key: "8ff09d423c9b79596a1c63fe6d58dae1"
server_key_iv: "062ab7b58b2114e58aadc43f1f66401b"
server_key_version: "encryptionkey@0"
I20260928 06:31:16.076025 4679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:16.085425 4679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:16.087852 4679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:16.088016 4679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:16.092732 4679 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.190:36479
I20260928 06:31:16.092828 4746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.190:36479 every 8 connection(s)
I20260928 06:31:16.093163 4679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
I20260928 06:31:16.094401 4747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:16.097674 4747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc5064433d164e12b54242dc5cb206fd: Bootstrap starting.
I20260928 06:31:16.098414 4747 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc5064433d164e12b54242dc5cb206fd: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:16.098740 4747 log.cc:824] T 00000000000000000000000000000000 P fc5064433d164e12b54242dc5cb206fd: Log is configured to *not* fsync() on all Append() calls
I20260928 06:31:16.099575 4747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc5064433d164e12b54242dc5cb206fd: No bootstrap required, opened a new log
I20260928 06:31:16.101990 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 4679
I20260928 06:31:16.101926 4747 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc5064433d164e12b54242dc5cb206fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc5064433d164e12b54242dc5cb206fd" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36479 } }
I20260928 06:31:16.102092 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/master-0/wal/instance
I20260928 06:31:16.102119 4747 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc5064433d164e12b54242dc5cb206fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:16.102159 4747 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc5064433d164e12b54242dc5cb206fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc5064433d164e12b54242dc5cb206fd, State: Initialized, Role: FOLLOWER
I20260928 06:31:16.102315 4747 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc5064433d164e12b54242dc5cb206fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc5064433d164e12b54242dc5cb206fd" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36479 } }
I20260928 06:31:16.102360 23430 external_mini_cluster.cc:1428] Setting key a5dab76816b15373403649d44772f0cb
I20260928 06:31:16.102396 4747 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc5064433d164e12b54242dc5cb206fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:31:16.102428 4747 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc5064433d164e12b54242dc5cb206fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:31:16.102473 4747 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc5064433d164e12b54242dc5cb206fd [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:16.103852 4747 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc5064433d164e12b54242dc5cb206fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc5064433d164e12b54242dc5cb206fd" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36479 } }
I20260928 06:31:16.103971 4747 leader_election.cc:302] T 00000000000000000000000000000000 P fc5064433d164e12b54242dc5cb206fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc5064433d164e12b54242dc5cb206fd; no voters:
I20260928 06:31:16.104156 4747 leader_election.cc:288] T 00000000000000000000000000000000 P fc5064433d164e12b54242dc5cb206fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:31:16.104187 4752 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc5064433d164e12b54242dc5cb206fd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:31:16.104346 4752 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc5064433d164e12b54242dc5cb206fd [term 1 LEADER]: Becoming Leader. State: Replica: fc5064433d164e12b54242dc5cb206fd, State: Running, Role: LEADER
I20260928 06:31:16.104415 4747 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc5064433d164e12b54242dc5cb206fd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:31:16.104439 4752 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc5064433d164e12b54242dc5cb206fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc5064433d164e12b54242dc5cb206fd" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36479 } }
I20260928 06:31:16.104799 4752 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc5064433d164e12b54242dc5cb206fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc5064433d164e12b54242dc5cb206fd. Latest consensus state: current_term: 1 leader_uuid: "fc5064433d164e12b54242dc5cb206fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc5064433d164e12b54242dc5cb206fd" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36479 } } }
I20260928 06:31:16.104875 4752 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc5064433d164e12b54242dc5cb206fd [sys.catalog]: This master's current role is: LEADER
I20260928 06:31:16.105033 4753 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc5064433d164e12b54242dc5cb206fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc5064433d164e12b54242dc5cb206fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc5064433d164e12b54242dc5cb206fd" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36479 } } }
I20260928 06:31:16.105116 4753 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc5064433d164e12b54242dc5cb206fd [sys.catalog]: This master's current role is: LEADER
W20260928 06:31:16.106283 4768 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fc5064433d164e12b54242dc5cb206fd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:31:16.106339 4768 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:31:16.106598 4758 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:31:16.106885 4758 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:31:16.108700 4758 catalog_manager.cc:1347] Generated new cluster ID: 48889f2c3c5e41d0b73452627b7870a1
I20260928 06:31:16.108750 4758 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:31:16 dist-test-slave-d59m krb5kdc[4666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:31:16.115468 4751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:16.102869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33962 (local address 127.22.225.190:36479)
0928 06:31:16.103061 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:16.103071 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:16.103117 (+ 46us) server_negotiation.cc:407] Connection header received
0928 06:31:16.103253 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:16.103263 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:16.103322 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:16.103534 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:16.104133 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:16.106258 (+ 2125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:16.107437 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:16.107966 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:16.108438 (+ 472us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:16.111027 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:16.111032 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:16.111032 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:16.111045 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:16.113215 (+ 2170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:16.113647 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:16.113650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:16.113651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:16.113776 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:16.113993 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:16.113994 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:16.113994 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:16.114315 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:16.114949 (+ 634us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:16.115032 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 06:31:16.115212 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":139,"thread_start_us":104,"threads_started":1}
I20260928 06:31:16.117616 4758 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:31:16.118330 4758 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:31:16.121224 4758 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc5064433d164e12b54242dc5cb206fd: Generated new TSK 0
I20260928 06:31:16.121381 4758 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.225.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab.
Entry for principal kudu/127.22.225.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab.
I20260928 06:31:16.157070 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.129:0
--local_ip_for_outbound_sockets=127.22.225.129
--webserver_interface=127.22.225.129
--webserver_port=0
--tserver_master_addrs=127.22.225.190:36479
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab
--principal=kudu/127.22.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:37031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:31:16.249727 4773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:16.250010 4773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:16.250048 4773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:16.251782 4773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:16.251838 4773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:16.251883 4773 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.129
I20260928 06:31:16.254048 4773 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:37031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab
--principal=kudu/127.22.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.225.129
--webserver_port=0
--tserver_master_addrs=127.22.225.190:36479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4773
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.129
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:31:16.254318 4773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:16.254662 4773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:16.260977 4779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:16.261090 4773 server_base.cc:1034] running on GCE node
W20260928 06:31:16.260982 4781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:16.261358 4778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:16.261575 4773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:16.261838 4773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:16.263017 4773 hybrid_clock.cc:648] HybridClock initialized: now 1790577076262994 us; error 35 us; skew 500 ppm
Sep 28 06:31:16 dist-test-slave-d59m krb5kdc[4666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577076, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:16.266608 4773 init.cc:377] Logged in from keytab as kudu/127.22.225.129@KRBTEST.COM (short username kudu)
I20260928 06:31:16.267287 4773 webserver.cc:460] Webserver started at http://127.22.225.129:45539/ using document root <none> and password file <none>
I20260928 06:31:16.267562 4773 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:16.267630 4773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:16.267768 4773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:16.269256 4773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-0/data/instance:
uuid: "25bfab18c92d4f6cb8727ef443a59867"
format_stamp: "Formatted at 2026-09-28 06:31:16 on dist-test-slave-d59m"
server_key: "3c19ffd404f956b5dcc1ca39f58ab04f"
server_key_iv: "4e0b9e4fdda099feea9abffca4d91885"
server_key_version: "encryptionkey@0"
I20260928 06:31:16.269667 4773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-0/wal/instance:
uuid: "25bfab18c92d4f6cb8727ef443a59867"
format_stamp: "Formatted at 2026-09-28 06:31:16 on dist-test-slave-d59m"
server_key: "3c19ffd404f956b5dcc1ca39f58ab04f"
server_key_iv: "4e0b9e4fdda099feea9abffca4d91885"
server_key_version: "encryptionkey@0"
I20260928 06:31:16.271598 4773 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:31:16.272537 4788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:16.272801 4773 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:16.272886 4773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-0/wal
uuid: "25bfab18c92d4f6cb8727ef443a59867"
format_stamp: "Formatted at 2026-09-28 06:31:16 on dist-test-slave-d59m"
server_key: "3c19ffd404f956b5dcc1ca39f58ab04f"
server_key_iv: "4e0b9e4fdda099feea9abffca4d91885"
server_key_version: "encryptionkey@0"
I20260928 06:31:16.272965 4773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:16.290395 4773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:16.292912 4773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:16.293090 4773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:16.293365 4773 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:31:16.293655 4773 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:31:16.293709 4773 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:16.293742 4773 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:31:16.293771 4773 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:16.300638 4773 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.129:34211
I20260928 06:31:16.300701 4901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.129:34211 every 8 connection(s)
I20260928 06:31:16.301120 4773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
I20260928 06:31:16.303258 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 4773
I20260928 06:31:16.303383 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-0/wal/instance
I20260928 06:31:16.304066 23430 external_mini_cluster.cc:1428] Setting key 1633d5fe2ed37c9ff6ebe013dfa09a65
Sep 28 06:31:16 dist-test-slave-d59m krb5kdc[4666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577076, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.129@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:31:16.312750 4751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:16.301920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:50169 (local address 127.22.225.190:36479)
0928 06:31:16.301968 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:16.301970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:16.302244 (+ 274us) server_negotiation.cc:407] Connection header received
0928 06:31:16.302802 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:16.302803 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:16.302818 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:16.302884 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:16.303973 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:16.304832 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:16.306578 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:16.306785 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:16.306872 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:16.309711 (+ 2839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:16.309713 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:16.309713 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:16.309721 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:16.311208 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:16.311666 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:16.311667 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:16.311668 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:16.311732 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:16.312008 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:16.312009 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:16.312010 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:16.312139 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:16.312212 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:16.312533 (+ 321us) server_negotiation.cc:299] Negotiation successful
0928 06:31:16.312653 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:31:16.312973 4904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:16.301915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36479 (local address 127.22.225.129:50169)
0928 06:31:16.302093 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:16.302139 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:16.302686 (+ 547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:16.302946 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:16.302950 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:16.303610 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:16.303849 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:16.303855 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:16.304940 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:16.304941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:16.306459 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:16.306464 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:16.306884 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:16.306885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:16.307032 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:16.307582 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:16.307593 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:16.309614 (+ 2021us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:16.311315 (+ 1701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:16.311324 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:16.311326 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:16.311570 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:16.311805 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:16.311806 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:16.311806 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:16.311930 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:16.312215 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:16.312223 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:16.312379 (+ 156us) client_negotiation.cc:769] Sending connection context
0928 06:31:16.312496 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 06:31:16.312707 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":105,"threads_started":1}
I20260928 06:31:16.313520 4902 heartbeater.cc:347] Connected to a master server at 127.22.225.190:36479
I20260928 06:31:16.313630 4902 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:16.313822 4902 heartbeater.cc:511] Master 127.22.225.190:36479 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.22.225.130@KRBTEST.COM; defaulting to no policy
I20260928 06:31:16.314606 4711 ts_manager.cc:194] Registered new tserver with Master: 25bfab18c92d4f6cb8727ef443a59867 (127.22.225.129:34211)
I20260928 06:31:16.315407 4711 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.225.129@KRBTEST.COM'} at 127.22.225.129:50169
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab.
Entry for principal kudu/127.22.225.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab.
I20260928 06:31:16.335980 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.130:0
--local_ip_for_outbound_sockets=127.22.225.130
--webserver_interface=127.22.225.130
--webserver_port=0
--tserver_master_addrs=127.22.225.190:36479
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab
--principal=kudu/127.22.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:37031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:31:16.435659 4907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:16.435940 4907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:16.435981 4907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:16.437459 4907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:16.437525 4907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:16.437575 4907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.130
I20260928 06:31:16.439621 4907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:37031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab
--principal=kudu/127.22.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.225.130
--webserver_port=0
--tserver_master_addrs=127.22.225.190:36479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.130
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:31:16.439911 4907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:16.440161 4907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:16.445382 4913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:16.445407 4915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:16.445909 4907 server_base.cc:1034] running on GCE node
W20260928 06:31:16.448161 4912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:16.448418 4907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:16.448727 4907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:16.449915 4907 hybrid_clock.cc:648] HybridClock initialized: now 1790577076449891 us; error 42 us; skew 500 ppm
Sep 28 06:31:16 dist-test-slave-d59m krb5kdc[4666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577076, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:16.452680 4907 init.cc:377] Logged in from keytab as kudu/127.22.225.130@KRBTEST.COM (short username kudu)
I20260928 06:31:16.453146 4907 webserver.cc:460] Webserver started at http://127.22.225.130:43959/ using document root <none> and password file <none>
I20260928 06:31:16.453400 4907 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:16.453445 4907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:16.453603 4907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:16.454923 4907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-1/data/instance:
uuid: "cf5ca4308f864408828089cc6a5532af"
format_stamp: "Formatted at 2026-09-28 06:31:16 on dist-test-slave-d59m"
server_key: "3af42ba6c7a3c7c037e2af04dc35f717"
server_key_iv: "30a0f8eeb009c9dce6d5b421f7b004f6"
server_key_version: "encryptionkey@0"
I20260928 06:31:16.455309 4907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-1/wal/instance:
uuid: "cf5ca4308f864408828089cc6a5532af"
format_stamp: "Formatted at 2026-09-28 06:31:16 on dist-test-slave-d59m"
server_key: "3af42ba6c7a3c7c037e2af04dc35f717"
server_key_iv: "30a0f8eeb009c9dce6d5b421f7b004f6"
server_key_version: "encryptionkey@0"
I20260928 06:31:16.456993 4907 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.003s
I20260928 06:31:16.457823 4922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:16.458003 4907 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:16.458081 4907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-1/wal
uuid: "cf5ca4308f864408828089cc6a5532af"
format_stamp: "Formatted at 2026-09-28 06:31:16 on dist-test-slave-d59m"
server_key: "3af42ba6c7a3c7c037e2af04dc35f717"
server_key_iv: "30a0f8eeb009c9dce6d5b421f7b004f6"
server_key_version: "encryptionkey@0"
I20260928 06:31:16.458149 4907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:16.472275 4907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:16.474723 4907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:16.474898 4907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:16.475147 4907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:31:16.475482 4907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:31:16.475533 4907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:16.475577 4907 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:31:16.475612 4907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:16.482400 4907 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.130:41535
I20260928 06:31:16.482440 5035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.130:41535 every 8 connection(s)
I20260928 06:31:16.482805 4907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
Sep 28 06:31:16 dist-test-slave-d59m krb5kdc[4666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577076, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.130@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:31:16.491420 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 4907
I20260928 06:31:16.491518 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-1/wal/instance
I20260928 06:31:16.491717 23430 external_mini_cluster.cc:1428] Setting key 10de018ced89edea1dc8852ef61fdd3d
I20260928 06:31:16.493202 4751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:16.483623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:44061 (local address 127.22.225.190:36479)
0928 06:31:16.483687 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:16.483689 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:16.483894 (+ 205us) server_negotiation.cc:407] Connection header received
0928 06:31:16.484476 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:16.484478 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:16.484493 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:16.484546 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:16.485657 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:16.486237 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:16.487248 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:16.487458 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:16.487543 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:16.489986 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:16.489987 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:16.489988 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:16.489995 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:16.491574 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:16.492096 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:16.492097 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:16.492098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:16.492163 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:16.492449 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:16.492450 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:16.492451 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:16.492621 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:16.492712 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:16.493065 (+ 353us) server_negotiation.cc:299] Negotiation successful
0928 06:31:16.493139 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 06:31:16.493682 5038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:16.483605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36479 (local address 127.22.225.130:44061)
0928 06:31:16.483767 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:16.483806 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:16.484366 (+ 560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:16.484691 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:16.484694 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:16.485338 (+ 644us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:16.485557 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:16.485563 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:16.486331 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:16.486332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:16.487154 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:16.487156 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:16.487557 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:16.487558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:16.487680 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:16.488298 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:16.488309 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:16.489878 (+ 1569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:16.491668 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:16.491676 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:16.491678 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:16.492018 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:16.492231 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:16.492232 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:16.492232 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:16.492377 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:16.492716 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:16.492723 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:16.492904 (+ 181us) client_negotiation.cc:769] Sending connection context
0928 06:31:16.493036 (+ 132us) client_negotiation.cc:240] Negotiation successful
0928 06:31:16.493225 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":85,"threads_started":1}
I20260928 06:31:16.494218 5036 heartbeater.cc:347] Connected to a master server at 127.22.225.190:36479
I20260928 06:31:16.494338 5036 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:16.494551 5036 heartbeater.cc:511] Master 127.22.225.190:36479 requested a full tablet report, sending...
I20260928 06:31:16.494977 4711 ts_manager.cc:194] Registered new tserver with Master: cf5ca4308f864408828089cc6a5532af (127.22.225.130:41535)
I20260928 06:31:16.495543 4711 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.225.130@KRBTEST.COM'} at 127.22.225.130:44061
WARNING: no policy specified for kudu/127.22.225.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab.
Entry for principal kudu/127.22.225.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab.
I20260928 06:31:16.524140 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.131:0
--local_ip_for_outbound_sockets=127.22.225.131
--webserver_interface=127.22.225.131
--webserver_port=0
--tserver_master_addrs=127.22.225.190:36479
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab
--principal=kudu/127.22.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:37031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:31:16.618363 5041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:16.618744 5041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:16.618803 5041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:16.620510 5041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:16.620565 5041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:16.620625 5041 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.131
I20260928 06:31:16.622921 5041 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:37031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab
--principal=kudu/127.22.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.225.131
--webserver_port=0
--tserver_master_addrs=127.22.225.190:36479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5041
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.131
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:31:16.623200 5041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:16.623502 5041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:16.628902 5046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:16.628911 5047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:16.629011 5049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:16.629316 5041 server_base.cc:1034] running on GCE node
I20260928 06:31:16.629525 5041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:16.629794 5041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:16.630956 5041 hybrid_clock.cc:648] HybridClock initialized: now 1790577076630940 us; error 34 us; skew 500 ppm
Sep 28 06:31:16 dist-test-slave-d59m krb5kdc[4666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577076, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:16.633778 5041 init.cc:377] Logged in from keytab as kudu/127.22.225.131@KRBTEST.COM (short username kudu)
I20260928 06:31:16.634224 5041 webserver.cc:460] Webserver started at http://127.22.225.131:39043/ using document root <none> and password file <none>
I20260928 06:31:16.634474 5041 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:16.634541 5041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:16.634680 5041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:16.635748 5041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-2/data/instance:
uuid: "87f7f1fcfb1f4202973d15f5827e8d13"
format_stamp: "Formatted at 2026-09-28 06:31:16 on dist-test-slave-d59m"
server_key: "0092f1137bc07d026e369fbaaa9094b9"
server_key_iv: "180de78533ad148654ac8275a18e1361"
server_key_version: "encryptionkey@0"
I20260928 06:31:16.636116 5041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-2/wal/instance:
uuid: "87f7f1fcfb1f4202973d15f5827e8d13"
format_stamp: "Formatted at 2026-09-28 06:31:16 on dist-test-slave-d59m"
server_key: "0092f1137bc07d026e369fbaaa9094b9"
server_key_iv: "180de78533ad148654ac8275a18e1361"
server_key_version: "encryptionkey@0"
I20260928 06:31:16.637776 5041 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:31:16.638674 5056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:16.638865 5041 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:16.638975 5041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-2/wal
uuid: "87f7f1fcfb1f4202973d15f5827e8d13"
format_stamp: "Formatted at 2026-09-28 06:31:16 on dist-test-slave-d59m"
server_key: "0092f1137bc07d026e369fbaaa9094b9"
server_key_iv: "180de78533ad148654ac8275a18e1361"
server_key_version: "encryptionkey@0"
I20260928 06:31:16.639062 5041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:16.650941 5041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:16.653563 5041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:16.653712 5041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:16.653944 5041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:31:16.654256 5041 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:31:16.654290 5041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:16.654328 5041 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:31:16.654356 5041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:16.661701 5041 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.131:38549
I20260928 06:31:16.661767 5169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.131:38549 every 8 connection(s)
I20260928 06:31:16.662220 5041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
Sep 28 06:31:16 dist-test-slave-d59m krb5kdc[4666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577076, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.131@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:31:16.669516 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 5041
I20260928 06:31:16.669608 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-2/wal/instance
I20260928 06:31:16.669806 23430 external_mini_cluster.cc:1428] Setting key 2ab8db3951ea5728441cb59080babe93
I20260928 06:31:16.672760 4751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:16.663055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:41687 (local address 127.22.225.190:36479)
0928 06:31:16.663112 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:16.663114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:16.663331 (+ 217us) server_negotiation.cc:407] Connection header received
0928 06:31:16.664120 (+ 789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:16.664121 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:16.664130 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:16.664202 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:16.665530 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:16.666075 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:16.667101 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:16.667317 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:16.667399 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:16.669710 (+ 2311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:16.669712 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:16.669712 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:16.669720 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:16.671297 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:16.671728 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:16.671729 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:16.671729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:16.671794 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:16.672050 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:16.672051 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:16.672051 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:16.672183 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:16.672258 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:16.672546 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 06:31:16.672661 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 06:31:16.673154 5172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:16.663003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36479 (local address 127.22.225.131:41687)
0928 06:31:16.663181 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:16.663226 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:16.664008 (+ 782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:16.664266 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:16.664271 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:16.665066 (+ 795us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:16.665393 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:16.665401 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:16.666194 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:16.666195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:16.667007 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:16.667009 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:16.667413 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:16.667414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:16.667544 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:16.668107 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:16.668116 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:16.669611 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:16.671390 (+ 1779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:16.671398 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:16.671401 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:16.671636 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:16.671855 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:16.671856 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:16.671857 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:16.671982 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:16.672249 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:16.672257 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:16.672412 (+ 155us) client_negotiation.cc:769] Sending connection context
0928 06:31:16.672524 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 06:31:16.672737 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":96,"threads_started":1}
I20260928 06:31:16.673628 5170 heartbeater.cc:347] Connected to a master server at 127.22.225.190:36479
I20260928 06:31:16.673744 5170 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:16.673933 5170 heartbeater.cc:511] Master 127.22.225.190:36479 requested a full tablet report, sending...
I20260928 06:31:16.674322 4711 ts_manager.cc:194] Registered new tserver with Master: 87f7f1fcfb1f4202973d15f5827e8d13 (127.22.225.131:38549)
I20260928 06:31:16.674829 4711 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.225.131@KRBTEST.COM'} at 127.22.225.131:41687
I20260928 06:31:16.681421 23430 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:31:16.690804 4751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:16.682800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33964 (local address 127.22.225.190:36479)
0928 06:31:16.682853 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:16.682855 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:16.682974 (+ 119us) server_negotiation.cc:407] Connection header received
0928 06:31:16.683059 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:16.683060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:16.683071 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:16.683137 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:16.683846 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:16.684631 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:16.685715 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:16.685993 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:16.686033 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:16.687496 (+ 1463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:16.687497 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:16.687498 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:16.687506 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:16.689516 (+ 2010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:16.689936 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:16.689937 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:16.689938 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:16.689994 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:16.690213 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:16.690214 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:16.690214 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:16.690381 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:16.690486 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:16.690600 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 06:31:16.690687 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 06:31:16.692646 4711 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33964:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:31:16.694821 4711 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:31:16.703275 5185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:16.697753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.130:41535 (local address 127.0.0.1:57628)
0928 06:31:16.698270 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:16.698278 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:16.698321 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:16.698612 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:16.698615 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:16.698617 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:16.698695 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:16.698700 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:16.700329 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:16.700330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:16.701663 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:16.701665 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:16.702961 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:16.702962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:16.703048 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:16.703062 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 06:31:16.703128 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 06:31:16.703191 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":74,"threads_started":1}
I20260928 06:31:16.703685 5186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:16.697901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57628 (local address 127.22.225.130:41535)
0928 06:31:16.698083 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:16.698089 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:16.698334 (+ 245us) server_negotiation.cc:407] Connection header received
0928 06:31:16.698389 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:16.698393 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:16.698410 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:16.698490 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:16.698788 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:16.700114 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:16.701763 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:16.702801 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:16.702867 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:16.702930 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:16.703557 (+ 627us) server_negotiation.cc:299] Negotiation successful
0928 06:31:16.703603 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":112,"threads_started":1}
I20260928 06:31:16.704233 5182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:16.697556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.131:38549 (local address 127.0.0.1:37092)
0928 06:31:16.698109 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:16.698119 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:16.698193 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:16.698686 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:16.698687 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:16.698688 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:16.698729 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:16.698731 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:16.701638 (+ 2907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:16.701639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:16.702760 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:16.702763 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:16.703929 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:16.703930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:16.703999 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:16.704049 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 06:31:16.704099 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 06:31:16.704139 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":92,"threads_started":1}
I20260928 06:31:16.704532 5184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:16.697669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37092 (local address 127.22.225.131:38549)
0928 06:31:16.698451 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:16.698456 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:16.698465 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:31:16.698505 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:16.698507 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:16.698521 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:16.698598 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:16.700366 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:16.701512 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:16.702871 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:16.703752 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:16.703833 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:16.703889 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:16.704397 (+ 508us) server_negotiation.cc:299] Negotiation successful
0928 06:31:16.704445 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":739,"thread_start_us":75,"threads_started":1}
I20260928 06:31:16.704460 4970 tablet_service.cc:1467] Processing CreateTablet for tablet ddcad719a8944ed39e43c11cd2015f38 (DEFAULT_TABLE table=test-table [id=92c37192349e41e4b740a396b71bb5dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:31:16.704907 4970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddcad719a8944ed39e43c11cd2015f38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:16.705497 5181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:16.697518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.129:34211 (local address 127.0.0.1:33222)
0928 06:31:16.697680 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:16.697725 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:16.697800 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:16.699044 (+ 1244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:16.699045 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:16.699046 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:16.699102 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:16.699103 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:16.702906 (+ 3803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:16.702907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:16.704141 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:16.704144 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:16.705215 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:16.705216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:16.705292 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:16.705296 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:31:16.705362 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 06:31:16.705411 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":57,"threads_started":1}
I20260928 06:31:16.705513 5183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:16.697615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33222 (local address 127.22.225.129:34211)
0928 06:31:16.698877 (+ 1262us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:16.698882 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:16.698891 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:31:16.698929 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:16.698931 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:16.698946 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:16.699173 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:16.699247 (+ 74us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:16.700251 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:16.704245 (+ 3994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:16.705099 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:16.705209 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:16.705261 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:16.705376 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 06:31:16.705422 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1213,"thread_start_us":97,"threads_started":1}
I20260928 06:31:16.705403 5104 tablet_service.cc:1467] Processing CreateTablet for tablet ddcad719a8944ed39e43c11cd2015f38 (DEFAULT_TABLE table=test-table [id=92c37192349e41e4b740a396b71bb5dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:31:16.705765 5104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddcad719a8944ed39e43c11cd2015f38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:16.706247 4836 tablet_service.cc:1467] Processing CreateTablet for tablet ddcad719a8944ed39e43c11cd2015f38 (DEFAULT_TABLE table=test-table [id=92c37192349e41e4b740a396b71bb5dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:31:16.706573 4836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddcad719a8944ed39e43c11cd2015f38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:16.708242 5187 tablet_bootstrap.cc:492] T ddcad719a8944ed39e43c11cd2015f38 P 87f7f1fcfb1f4202973d15f5827e8d13: Bootstrap starting.
I20260928 06:31:16.708982 5187 tablet_bootstrap.cc:654] T ddcad719a8944ed39e43c11cd2015f38 P 87f7f1fcfb1f4202973d15f5827e8d13: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:16.709438 5188 tablet_bootstrap.cc:492] T ddcad719a8944ed39e43c11cd2015f38 P cf5ca4308f864408828089cc6a5532af: Bootstrap starting.
I20260928 06:31:16.709609 5187 log.cc:824] T ddcad719a8944ed39e43c11cd2015f38 P 87f7f1fcfb1f4202973d15f5827e8d13: Log is configured to *not* fsync() on all Append() calls
I20260928 06:31:16.710229 5188 tablet_bootstrap.cc:654] T ddcad719a8944ed39e43c11cd2015f38 P cf5ca4308f864408828089cc6a5532af: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:16.710454 5187 tablet_bootstrap.cc:492] T ddcad719a8944ed39e43c11cd2015f38 P 87f7f1fcfb1f4202973d15f5827e8d13: No bootstrap required, opened a new log
I20260928 06:31:16.710538 5187 ts_tablet_manager.cc:1397] T ddcad719a8944ed39e43c11cd2015f38 P 87f7f1fcfb1f4202973d15f5827e8d13: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 06:31:16.711071 5188 log.cc:824] T ddcad719a8944ed39e43c11cd2015f38 P cf5ca4308f864408828089cc6a5532af: Log is configured to *not* fsync() on all Append() calls
I20260928 06:31:16.711843 5188 tablet_bootstrap.cc:492] T ddcad719a8944ed39e43c11cd2015f38 P cf5ca4308f864408828089cc6a5532af: No bootstrap required, opened a new log
I20260928 06:31:16.711916 5188 ts_tablet_manager.cc:1397] T ddcad719a8944ed39e43c11cd2015f38 P cf5ca4308f864408828089cc6a5532af: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 06:31:16.712886 5190 tablet_bootstrap.cc:492] T ddcad719a8944ed39e43c11cd2015f38 P 25bfab18c92d4f6cb8727ef443a59867: Bootstrap starting.
I20260928 06:31:16.713107 5187 raft_consensus.cc:348] T ddcad719a8944ed39e43c11cd2015f38 P 87f7f1fcfb1f4202973d15f5827e8d13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25bfab18c92d4f6cb8727ef443a59867" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34211 } } peers { permanent_uuid: "87f7f1fcfb1f4202973d15f5827e8d13" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 38549 } } peers { permanent_uuid: "cf5ca4308f864408828089cc6a5532af" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41535 } }
I20260928 06:31:16.713323 5187 raft_consensus.cc:374] T ddcad719a8944ed39e43c11cd2015f38 P 87f7f1fcfb1f4202973d15f5827e8d13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:16.713366 5187 raft_consensus.cc:729] T ddcad719a8944ed39e43c11cd2015f38 P 87f7f1fcfb1f4202973d15f5827e8d13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87f7f1fcfb1f4202973d15f5827e8d13, State: Initialized, Role: FOLLOWER
I20260928 06:31:16.713531 5187 consensus_queue.cc:260] T ddcad719a8944ed39e43c11cd2015f38 P 87f7f1fcfb1f4202973d15f5827e8d13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25bfab18c92d4f6cb8727ef443a59867" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34211 } } peers { permanent_uuid: "87f7f1fcfb1f4202973d15f5827e8d13" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 38549 } } peers { permanent_uuid: "cf5ca4308f864408828089cc6a5532af" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41535 } }
I20260928 06:31:16.713840 5187 ts_tablet_manager.cc:1428] T ddcad719a8944ed39e43c11cd2015f38 P 87f7f1fcfb1f4202973d15f5827e8d13: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20260928 06:31:16.713845 5170 heartbeater.cc:503] Master 127.22.225.190:36479 was elected leader, sending a full tablet report...
I20260928 06:31:16.714352 5190 tablet_bootstrap.cc:654] T ddcad719a8944ed39e43c11cd2015f38 P 25bfab18c92d4f6cb8727ef443a59867: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:16.714565 5188 raft_consensus.cc:348] T ddcad719a8944ed39e43c11cd2015f38 P cf5ca4308f864408828089cc6a5532af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25bfab18c92d4f6cb8727ef443a59867" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34211 } } peers { permanent_uuid: "87f7f1fcfb1f4202973d15f5827e8d13" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 38549 } } peers { permanent_uuid: "cf5ca4308f864408828089cc6a5532af" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41535 } }
I20260928 06:31:16.714740 5188 raft_consensus.cc:374] T ddcad719a8944ed39e43c11cd2015f38 P cf5ca4308f864408828089cc6a5532af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:16.714793 5188 raft_consensus.cc:729] T ddcad719a8944ed39e43c11cd2015f38 P cf5ca4308f864408828089cc6a5532af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf5ca4308f864408828089cc6a5532af, State: Initialized, Role: FOLLOWER
I20260928 06:31:16.714926 5188 consensus_queue.cc:260] T ddcad719a8944ed39e43c11cd2015f38 P cf5ca4308f864408828089cc6a5532af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25bfab18c92d4f6cb8727ef443a59867" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34211 } } peers { permanent_uuid: "87f7f1fcfb1f4202973d15f5827e8d13" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 38549 } } peers { permanent_uuid: "cf5ca4308f864408828089cc6a5532af" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41535 } }
I20260928 06:31:16.715240 5036 heartbeater.cc:503] Master 127.22.225.190:36479 was elected leader, sending a full tablet report...
I20260928 06:31:16.715248 5190 log.cc:824] T ddcad719a8944ed39e43c11cd2015f38 P 25bfab18c92d4f6cb8727ef443a59867: Log is configured to *not* fsync() on all Append() calls
I20260928 06:31:16.715534 5188 ts_tablet_manager.cc:1428] T ddcad719a8944ed39e43c11cd2015f38 P cf5ca4308f864408828089cc6a5532af: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260928 06:31:16.716162 5190 tablet_bootstrap.cc:492] T ddcad719a8944ed39e43c11cd2015f38 P 25bfab18c92d4f6cb8727ef443a59867: No bootstrap required, opened a new log
I20260928 06:31:16.716229 5190 ts_tablet_manager.cc:1397] T ddcad719a8944ed39e43c11cd2015f38 P 25bfab18c92d4f6cb8727ef443a59867: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 06:31:16.718505 5190 raft_consensus.cc:348] T ddcad719a8944ed39e43c11cd2015f38 P 25bfab18c92d4f6cb8727ef443a59867 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25bfab18c92d4f6cb8727ef443a59867" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34211 } } peers { permanent_uuid: "87f7f1fcfb1f4202973d15f5827e8d13" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 38549 } } peers { permanent_uuid: "cf5ca4308f864408828089cc6a5532af" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41535 } }
I20260928 06:31:16.718623 5190 raft_consensus.cc:374] T ddcad719a8944ed39e43c11cd2015f38 P 25bfab18c92d4f6cb8727ef443a59867 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:16.718643 5190 raft_consensus.cc:729] T ddcad719a8944ed39e43c11cd2015f38 P 25bfab18c92d4f6cb8727ef443a59867 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25bfab18c92d4f6cb8727ef443a59867, State: Initialized, Role: FOLLOWER
I20260928 06:31:16.718762 5190 consensus_queue.cc:260] T ddcad719a8944ed39e43c11cd2015f38 P 25bfab18c92d4f6cb8727ef443a59867 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25bfab18c92d4f6cb8727ef443a59867" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34211 } } peers { permanent_uuid: "87f7f1fcfb1f4202973d15f5827e8d13" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 38549 } } peers { permanent_uuid: "cf5ca4308f864408828089cc6a5532af" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41535 } }
I20260928 06:31:16.719035 4902 heartbeater.cc:503] Master 127.22.225.190:36479 was elected leader, sending a full tablet report...
I20260928 06:31:16.719269 5190 ts_tablet_manager.cc:1428] T ddcad719a8944ed39e43c11cd2015f38 P 25bfab18c92d4f6cb8727ef443a59867: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260928 06:31:16.733515 5037 tablet.cc:2369] T ddcad719a8944ed39e43c11cd2015f38 P cf5ca4308f864408828089cc6a5532af: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:31:16.750083 5195 raft_consensus.cc:482] T ddcad719a8944ed39e43c11cd2015f38 P 25bfab18c92d4f6cb8727ef443a59867 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:31:16.750205 5195 raft_consensus.cc:504] T ddcad719a8944ed39e43c11cd2015f38 P 25bfab18c92d4f6cb8727ef443a59867 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25bfab18c92d4f6cb8727ef443a59867" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34211 } } peers { permanent_uuid: "87f7f1fcfb1f4202973d15f5827e8d13" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 38549 } } peers { permanent_uuid: "cf5ca4308f864408828089cc6a5532af" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41535 } }
I20260928 06:31:16.750667 5195 leader_election.cc:288] T ddcad719a8944ed39e43c11cd2015f38 P 25bfab18c92d4f6cb8727ef443a59867 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87f7f1fcfb1f4202973d15f5827e8d13 (127.22.225.131:38549), cf5ca4308f864408828089cc6a5532af (127.22.225.130:41535)
I20260928 06:31:16.754156 4904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:16.750830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.131:38549 (local address 127.22.225.129:52955)
0928 06:31:16.750887 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:16.750900 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:16.750946 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:16.751223 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:16.751223 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:16.751225 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:16.751319 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:16.751321 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:16.752160 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:16.752161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:16.753131 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:16.753133 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:16.753886 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:16.753886 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:16.753966 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:16.753969 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:31:16.754018 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 06:31:16.754047 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":21}
I20260928 06:31:16.754155 5186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:16.751069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:45147 (local address 127.22.225.130:41535)
0928 06:31:16.751136 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:16.751139 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:16.751200 (+ 61us) server_negotiation.cc:407] Connection header received
0928 06:31:16.751288 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:16.751290 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:16.751304 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:16.751369 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:16.751529 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:16.752010 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:16.753265 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:16.753686 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:16.753792 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:16.753801 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:16.754018 (+ 217us) server_negotiation.cc:299] Negotiation successful
0928 06:31:16.754076 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 06:31:16.754253 5184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:16.750928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:52955 (local address 127.22.225.131:38549)
0928 06:31:16.751063 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:16.751065 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:16.751076 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:31:16.751109 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:16.751110 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:16.751120 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:16.751203 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:16.751392 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:16.752068 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:16.753212 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:16.753803 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:16.753890 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:16.753900 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:16.754128 (+ 228us) server_negotiation.cc:299] Negotiation successful
0928 06:31:16.754171 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260928 06:31:16.754386 5196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:16.750952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.130:41535 (local address 127.22.225.129:45147)
0928 06:31:16.751135 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:16.751146 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:16.751215 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:16.751393 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:16.751393 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:16.751395 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:16.751462 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:16.751463 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:16.752088 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:16.752090 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:16.753156 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:16.753160 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:16.753791 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:16.753792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:16.753881 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:16.753885 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:31:16.753985 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 06:31:16.754037 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":54,"thread_start_us":106,"threads_started":1}
I20260928 06:31:16.754649 5124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddcad719a8944ed39e43c11cd2015f38" candidate_uuid: "25bfab18c92d4f6cb8727ef443a59867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87f7f1fcfb1f4202973d15f5827e8d13" is_pre_election: true
I20260928 06:31:16.754833 5124 raft_consensus.cc:2454] T ddcad719a8944ed39e43c11cd2015f38 P 87f7f1fcfb1f4202973d15f5827e8d13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25bfab18c92d4f6cb8727ef443a59867 in term 0.
I20260928 06:31:16.755069 4990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddcad719a8944ed39e43c11cd2015f38" candidate_uuid: "25bfab18c92d4f6cb8727ef443a59867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf5ca4308f864408828089cc6a5532af" is_pre_election: true
I20260928 06:31:16.755101 4792 leader_election.cc:302] T ddcad719a8944ed39e43c11cd2015f38 P 25bfab18c92d4f6cb8727ef443a59867 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25bfab18c92d4f6cb8727ef443a59867, 87f7f1fcfb1f4202973d15f5827e8d13; no voters:
I20260928 06:31:16.755236 4990 raft_consensus.cc:2454] T ddcad719a8944ed39e43c11cd2015f38 P cf5ca4308f864408828089cc6a5532af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25bfab18c92d4f6cb8727ef443a59867 in term 0.
I20260928 06:31:16.755252 5195 raft_consensus.cc:2781] T ddcad719a8944ed39e43c11cd2015f38 P 25bfab18c92d4f6cb8727ef443a59867 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:31:16.755297 5195 raft_consensus.cc:482] T ddcad719a8944ed39e43c11cd2015f38 P 25bfab18c92d4f6cb8727ef443a59867 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:31:16.755330 5195 raft_consensus.cc:3037] T ddcad719a8944ed39e43c11cd2015f38 P 25bfab18c92d4f6cb8727ef443a59867 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:16.756039 5195 raft_consensus.cc:504] T ddcad719a8944ed39e43c11cd2015f38 P 25bfab18c92d4f6cb8727ef443a59867 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25bfab18c92d4f6cb8727ef443a59867" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34211 } } peers { permanent_uuid: "87f7f1fcfb1f4202973d15f5827e8d13" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 38549 } } peers { permanent_uuid: "cf5ca4308f864408828089cc6a5532af" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41535 } }
I20260928 06:31:16.756181 5195 leader_election.cc:288] T ddcad719a8944ed39e43c11cd2015f38 P 25bfab18c92d4f6cb8727ef443a59867 [CANDIDATE]: Term 1 election: Requested vote from peers 87f7f1fcfb1f4202973d15f5827e8d13 (127.22.225.131:38549), cf5ca4308f864408828089cc6a5532af (127.22.225.130:41535)
I20260928 06:31:16.756333 5124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddcad719a8944ed39e43c11cd2015f38" candidate_uuid: "25bfab18c92d4f6cb8727ef443a59867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87f7f1fcfb1f4202973d15f5827e8d13"
I20260928 06:31:16.756357 4990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddcad719a8944ed39e43c11cd2015f38" candidate_uuid: "25bfab18c92d4f6cb8727ef443a59867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf5ca4308f864408828089cc6a5532af"
I20260928 06:31:16.756425 5124 raft_consensus.cc:3037] T ddcad719a8944ed39e43c11cd2015f38 P 87f7f1fcfb1f4202973d15f5827e8d13 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:16.756453 4990 raft_consensus.cc:3037] T ddcad719a8944ed39e43c11cd2015f38 P cf5ca4308f864408828089cc6a5532af [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:16.757254 4990 raft_consensus.cc:2454] T ddcad719a8944ed39e43c11cd2015f38 P cf5ca4308f864408828089cc6a5532af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25bfab18c92d4f6cb8727ef443a59867 in term 1.
I20260928 06:31:16.757421 4792 leader_election.cc:302] T ddcad719a8944ed39e43c11cd2015f38 P 25bfab18c92d4f6cb8727ef443a59867 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25bfab18c92d4f6cb8727ef443a59867, cf5ca4308f864408828089cc6a5532af; no voters:
I20260928 06:31:16.757471 5124 raft_consensus.cc:2454] T ddcad719a8944ed39e43c11cd2015f38 P 87f7f1fcfb1f4202973d15f5827e8d13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25bfab18c92d4f6cb8727ef443a59867 in term 1.
I20260928 06:31:16.757527 5195 raft_consensus.cc:2781] T ddcad719a8944ed39e43c11cd2015f38 P 25bfab18c92d4f6cb8727ef443a59867 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:31:16.757683 5195 raft_consensus.cc:686] T ddcad719a8944ed39e43c11cd2015f38 P 25bfab18c92d4f6cb8727ef443a59867 [term 1 LEADER]: Becoming Leader. State: Replica: 25bfab18c92d4f6cb8727ef443a59867, State: Running, Role: LEADER
I20260928 06:31:16.757802 5195 consensus_queue.cc:237] T ddcad719a8944ed39e43c11cd2015f38 P 25bfab18c92d4f6cb8727ef443a59867 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25bfab18c92d4f6cb8727ef443a59867" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34211 } } peers { permanent_uuid: "87f7f1fcfb1f4202973d15f5827e8d13" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 38549 } } peers { permanent_uuid: "cf5ca4308f864408828089cc6a5532af" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41535 } }
I20260928 06:31:16.758610 4711 catalog_manager.cc:5495] T ddcad719a8944ed39e43c11cd2015f38 P 25bfab18c92d4f6cb8727ef443a59867 reported cstate change: term changed from 0 to 1, leader changed from <none> to 25bfab18c92d4f6cb8727ef443a59867 (127.22.225.129). New cstate: current_term: 1 leader_uuid: "25bfab18c92d4f6cb8727ef443a59867" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25bfab18c92d4f6cb8727ef443a59867" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "87f7f1fcfb1f4202973d15f5827e8d13" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 38549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf5ca4308f864408828089cc6a5532af" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41535 } health_report { overall_health: UNKNOWN } } }
I20260928 06:31:16.777807 5183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:16.775081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33228 (local address 127.22.225.129:34211)
0928 06:31:16.775129 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:16.775131 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:16.775143 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:31:16.775185 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:16.775186 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:16.775209 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:16.775293 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:31:16.775443 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:16.775927 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:16.776934 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:16.777189 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:16.777260 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:16.777435 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:31:16.777548 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:31:16.777659 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:16.777679 (+ 20us) server_negotiation.cc:299] Negotiation successful
0928 06:31:16.777699 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 06:31:16.779959 4990 raft_consensus.cc:1261] T ddcad719a8944ed39e43c11cd2015f38 P cf5ca4308f864408828089cc6a5532af [term 1 FOLLOWER]: Refusing update from remote peer 25bfab18c92d4f6cb8727ef443a59867: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:31:16.779980 5124 raft_consensus.cc:1261] T ddcad719a8944ed39e43c11cd2015f38 P 87f7f1fcfb1f4202973d15f5827e8d13 [term 1 FOLLOWER]: Refusing update from remote peer 25bfab18c92d4f6cb8727ef443a59867: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:31:16.780373 5197 consensus_queue.cc:1035] T ddcad719a8944ed39e43c11cd2015f38 P 25bfab18c92d4f6cb8727ef443a59867 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87f7f1fcfb1f4202973d15f5827e8d13" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 38549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:31:16.780514 5195 consensus_queue.cc:1035] T ddcad719a8944ed39e43c11cd2015f38 P 25bfab18c92d4f6cb8727ef443a59867 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf5ca4308f864408828089cc6a5532af" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:31:16.783075 5200 mvcc.cc:205] Tried to move back new op lower bound from 7334203706487357440 to 7334203706400595968. Current Snapshot: MvccSnapshot[applied={T|T < 7334203706487357440}]
I20260928 06:31:16.783846 5201 mvcc.cc:205] Tried to move back new op lower bound from 7334203706487357440 to 7334203706400595968. Current Snapshot: MvccSnapshot[applied={T|T < 7334203706487357440}]
I20260928 06:31:16.784391 5202 mvcc.cc:205] Tried to move back new op lower bound from 7334203706487357440 to 7334203706400595968. Current Snapshot: MvccSnapshot[applied={T|T < 7334203706487357440 or (T in {7334203706487357440})}]
I20260928 06:31:16.785488 4710 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:31:16.785590 4710 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:31:16.786334 4710 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fc5064433d164e12b54242dc5cb206fd: Sending DeleteTablet for 3 replicas of tablet ddcad719a8944ed39e43c11cd2015f38
I20260928 06:31:16.786698 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 4773
I20260928 06:31:16.786684 4836 tablet_service.cc:1514] Processing DeleteTablet for tablet ddcad719a8944ed39e43c11cd2015f38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:31:16 UTC) from {username='kudu', principal='kudu/127.22.225.190@KRBTEST.COM'} at 127.0.0.1:33222
I20260928 06:31:16.786782 5104 tablet_service.cc:1514] Processing DeleteTablet for tablet ddcad719a8944ed39e43c11cd2015f38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:31:16 UTC) from {username='kudu', principal='kudu/127.22.225.190@KRBTEST.COM'} at 127.0.0.1:37092
I20260928 06:31:16.786976 4970 tablet_service.cc:1514] Processing DeleteTablet for tablet ddcad719a8944ed39e43c11cd2015f38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:31:16 UTC) from {username='kudu', principal='kudu/127.22.225.190@KRBTEST.COM'} at 127.0.0.1:57628
I20260928 06:31:16.789661 5212 tablet_replica.cc:330] T ddcad719a8944ed39e43c11cd2015f38 P cf5ca4308f864408828089cc6a5532af: stopping tablet replica
I20260928 06:31:16.789776 5212 raft_consensus.cc:2229] T ddcad719a8944ed39e43c11cd2015f38 P cf5ca4308f864408828089cc6a5532af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:31:16.789893 5212 raft_consensus.cc:2258] T ddcad719a8944ed39e43c11cd2015f38 P cf5ca4308f864408828089cc6a5532af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:31:16.790123 5211 tablet_replica.cc:330] T ddcad719a8944ed39e43c11cd2015f38 P 87f7f1fcfb1f4202973d15f5827e8d13: stopping tablet replica
I20260928 06:31:16.790239 5211 raft_consensus.cc:2229] T ddcad719a8944ed39e43c11cd2015f38 P 87f7f1fcfb1f4202973d15f5827e8d13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:31:16.790356 5211 raft_consensus.cc:2258] T ddcad719a8944ed39e43c11cd2015f38 P 87f7f1fcfb1f4202973d15f5827e8d13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:31:16.791260 5211 ts_tablet_manager.cc:1905] T ddcad719a8944ed39e43c11cd2015f38 P 87f7f1fcfb1f4202973d15f5827e8d13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:31:16.791263 5212 ts_tablet_manager.cc:1905] T ddcad719a8944ed39e43c11cd2015f38 P cf5ca4308f864408828089cc6a5532af: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 06:31:16.793357 4697 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.225.129:34211 (error 108)
I20260928 06:31:16.793511 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 4907
I20260928 06:31:16.793816 5211 ts_tablet_manager.cc:1918] T ddcad719a8944ed39e43c11cd2015f38 P 87f7f1fcfb1f4202973d15f5827e8d13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:31:16.793840 5212 ts_tablet_manager.cc:1918] T ddcad719a8944ed39e43c11cd2015f38 P cf5ca4308f864408828089cc6a5532af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:31:16.793888 5211 log.cc:1196] T ddcad719a8944ed39e43c11cd2015f38 P 87f7f1fcfb1f4202973d15f5827e8d13: Deleting WAL directory at /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-2/wal/wals/ddcad719a8944ed39e43c11cd2015f38
I20260928 06:31:16.793898 5212 log.cc:1196] T ddcad719a8944ed39e43c11cd2015f38 P cf5ca4308f864408828089cc6a5532af: Deleting WAL directory at /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-1/wal/wals/ddcad719a8944ed39e43c11cd2015f38
I20260928 06:31:16.794209 5211 ts_tablet_manager.cc:1939] T ddcad719a8944ed39e43c11cd2015f38 P 87f7f1fcfb1f4202973d15f5827e8d13: Deleting consensus metadata
I20260928 06:31:16.794725 4698 catalog_manager.cc:4841] TS 87f7f1fcfb1f4202973d15f5827e8d13 (127.22.225.131:38549): tablet ddcad719a8944ed39e43c11cd2015f38 (table test-table [id=92c37192349e41e4b740a396b71bb5dd]) successfully deleted
I20260928 06:31:16.795223 5181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:16.794152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.129:34211 (local address 127.0.0.1:33242)
0928 06:31:16.795152 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:16.795195 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.129:34211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":978}
W20260928 06:31:16.795359 4697 catalog_manager.cc:4584] TS 25bfab18c92d4f6cb8727ef443a59867 (127.22.225.129:34211): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ddcad719a8944ed39e43c11cd2015f38: Network error: Client connection negotiation failed: client connection to 127.22.225.129:34211: connect: Connection refused (error 111)
I20260928 06:31:16.800383 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 5041
I20260928 06:31:16.800433 5181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:16.800314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.130:41535 (local address 127.0.0.1:57638)
0928 06:31:16.800369 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:16.800395 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.130:41535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 06:31:16.806443 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 4679
I20260928 06:31:16.812117 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:36479
--webserver_interface=127.22.225.190
--webserver_port=41111
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab
--principal=kudu/127.22.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:37031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.225.190:36479
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:31:16.911437 5214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:16.911651 5214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:16.911674 5214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:16.912915 5214 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:31:16.912959 5214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:16.912973 5214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:16.912986 5214 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:31:16.912998 5214 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:31:16.914510 5214 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:37031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.22.225.190:36479
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab
--principal=kudu/127.22.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:36479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.225.190
--webserver_port=41111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5214
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:31:16.914765 5214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:16.915020 5214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:16.933876 5221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:16.933884 5220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:16.933874 5223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:16.934418 5214 server_base.cc:1034] running on GCE node
I20260928 06:31:16.934783 5214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:16.935233 5214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:16.936434 5214 hybrid_clock.cc:648] HybridClock initialized: now 1790577076936405 us; error 41 us; skew 500 ppm
Sep 28 06:31:16 dist-test-slave-d59m krb5kdc[4666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577076, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:16.940233 5214 init.cc:377] Logged in from keytab as kudu/127.22.225.190@KRBTEST.COM (short username kudu)
I20260928 06:31:16.940981 5214 webserver.cc:460] Webserver started at http://127.22.225.190:41111/ using document root <none> and password file <none>
I20260928 06:31:16.941335 5214 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:16.941434 5214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:16.943532 5214 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 06:31:16.944411 5230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:16.944641 5214 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:31:16.944764 5214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/master-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/master-0/wal
uuid: "fc5064433d164e12b54242dc5cb206fd"
format_stamp: "Formatted at 2026-09-28 06:31:16 on dist-test-slave-d59m"
server_key: "8ff09d423c9b79596a1c63fe6d58dae1"
server_key_iv: "062ab7b58b2114e58aadc43f1f66401b"
server_key_version: "encryptionkey@0"
I20260928 06:31:16.945410 5214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:16.963351 5214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:16.965873 5214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:16.966022 5214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:16.970932 5214 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.190:36479
I20260928 06:31:16.970958 5282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.190:36479 every 8 connection(s)
I20260928 06:31:16.971478 5214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
I20260928 06:31:16.973219 5283 sys_catalog.cc:263] Verifying existing consensus state
I20260928 06:31:16.973893 5283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc5064433d164e12b54242dc5cb206fd: Bootstrap starting.
I20260928 06:31:16.978123 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 5214
I20260928 06:31:16.978511 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.129:34211
--local_ip_for_outbound_sockets=127.22.225.129
--tserver_master_addrs=127.22.225.190:36479
--webserver_port=45539
--webserver_interface=127.22.225.129
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab
--principal=kudu/127.22.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:37031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:31:16.983567 5283 log.cc:824] T 00000000000000000000000000000000 P fc5064433d164e12b54242dc5cb206fd: Log is configured to *not* fsync() on all Append() calls
I20260928 06:31:16.985770 5283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc5064433d164e12b54242dc5cb206fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:31:16.986114 5283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc5064433d164e12b54242dc5cb206fd: Bootstrap complete.
I20260928 06:31:16.988533 5283 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc5064433d164e12b54242dc5cb206fd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc5064433d164e12b54242dc5cb206fd" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36479 } }
I20260928 06:31:16.988883 5283 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc5064433d164e12b54242dc5cb206fd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc5064433d164e12b54242dc5cb206fd, State: Initialized, Role: FOLLOWER
I20260928 06:31:16.989112 5283 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc5064433d164e12b54242dc5cb206fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc5064433d164e12b54242dc5cb206fd" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36479 } }
I20260928 06:31:16.989214 5283 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc5064433d164e12b54242dc5cb206fd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:31:16.989290 5283 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc5064433d164e12b54242dc5cb206fd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:31:16.989347 5283 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc5064433d164e12b54242dc5cb206fd [term 1 FOLLOWER]: Advancing to term 2
I20260928 06:31:16.990414 5283 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc5064433d164e12b54242dc5cb206fd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc5064433d164e12b54242dc5cb206fd" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36479 } }
I20260928 06:31:16.990535 5283 leader_election.cc:302] T 00000000000000000000000000000000 P fc5064433d164e12b54242dc5cb206fd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc5064433d164e12b54242dc5cb206fd; no voters:
I20260928 06:31:16.990734 5283 leader_election.cc:288] T 00000000000000000000000000000000 P fc5064433d164e12b54242dc5cb206fd [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 06:31:16.990772 5287 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc5064433d164e12b54242dc5cb206fd [term 2 FOLLOWER]: Leader election won for term 2
I20260928 06:31:16.990974 5287 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc5064433d164e12b54242dc5cb206fd [term 2 LEADER]: Becoming Leader. State: Replica: fc5064433d164e12b54242dc5cb206fd, State: Running, Role: LEADER
I20260928 06:31:16.991109 5283 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc5064433d164e12b54242dc5cb206fd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:31:16.991127 5287 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc5064433d164e12b54242dc5cb206fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc5064433d164e12b54242dc5cb206fd" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36479 } }
I20260928 06:31:16.991422 5288 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc5064433d164e12b54242dc5cb206fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "fc5064433d164e12b54242dc5cb206fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc5064433d164e12b54242dc5cb206fd" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36479 } } }
I20260928 06:31:16.991524 5288 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc5064433d164e12b54242dc5cb206fd [sys.catalog]: This master's current role is: LEADER
I20260928 06:31:16.991693 5289 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc5064433d164e12b54242dc5cb206fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc5064433d164e12b54242dc5cb206fd. Latest consensus state: current_term: 2 leader_uuid: "fc5064433d164e12b54242dc5cb206fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc5064433d164e12b54242dc5cb206fd" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36479 } } }
I20260928 06:31:16.992660 5289 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc5064433d164e12b54242dc5cb206fd [sys.catalog]: This master's current role is: LEADER
I20260928 06:31:16.993693 5302 catalog_manager.cc:1259] Loaded cluster ID: 48889f2c3c5e41d0b73452627b7870a1
I20260928 06:31:16.993743 5302 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fc5064433d164e12b54242dc5cb206fd: loading cluster ID for follower catalog manager: success
W20260928 06:31:16.993940 5302 catalog_manager.cc:1576] T 00000000000000000000000000000000 P fc5064433d164e12b54242dc5cb206fd: acquiring CA information for follower catalog manager: Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG
W20260928 06:31:16.993979 5302 catalog_manager.cc:873] Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG: failed to prepare follower catalog manager, will retry
I20260928 06:31:16.994023 5291 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:31:16.994431 5291 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:31:16.994510 5291 catalog_manager.cc:1259] Loaded cluster ID: 48889f2c3c5e41d0b73452627b7870a1
I20260928 06:31:16.994540 5291 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:31:16.994742 5291 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 06:31:16.999832 5291 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 06:31:17.002784 5291 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:31:17.003072 5291 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fc5064433d164e12b54242dc5cb206fd: Loaded TSK: 0
I20260928 06:31:17.003381 5291 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 06:31:17.070209 5285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:17.070449 5285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:17.070528 5285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:17.071782 5285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:17.071852 5285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:17.071900 5285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.129
I20260928 06:31:17.073458 5285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:37031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab
--principal=kudu/127.22.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.129:34211
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.225.129
--webserver_port=45539
--tserver_master_addrs=127.22.225.190:36479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.129
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:31:17.073765 5285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:17.074076 5285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:17.079595 5313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:17.079728 5285 server_base.cc:1034] running on GCE node
W20260928 06:31:17.079963 5311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:17.080183 5310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:17.080442 5285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:17.080745 5285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:17.081913 5285 hybrid_clock.cc:648] HybridClock initialized: now 1790577077081886 us; error 46 us; skew 500 ppm
Sep 28 06:31:17 dist-test-slave-d59m krb5kdc[4666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577077, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:17.084789 5285 init.cc:377] Logged in from keytab as kudu/127.22.225.129@KRBTEST.COM (short username kudu)
I20260928 06:31:17.085248 5285 webserver.cc:460] Webserver started at http://127.22.225.129:45539/ using document root <none> and password file <none>
I20260928 06:31:17.085520 5285 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:17.085605 5285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:17.087088 5285 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 06:31:17.087839 5320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:17.088006 5285 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:31:17.088115 5285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-0/wal
uuid: "25bfab18c92d4f6cb8727ef443a59867"
format_stamp: "Formatted at 2026-09-28 06:31:16 on dist-test-slave-d59m"
server_key: "3c19ffd404f956b5dcc1ca39f58ab04f"
server_key_iv: "4e0b9e4fdda099feea9abffca4d91885"
server_key_version: "encryptionkey@0"
I20260928 06:31:17.088491 5285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:17.101927 5285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:17.104440 5285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:17.104635 5285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:17.104918 5285 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:31:17.105468 5327 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 06:31:17.106920 5285 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 06:31:17.106973 5285 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 06:31:17.107029 5285 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 06:31:17.107702 5285 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 06:31:17.107779 5285 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.001s
I20260928 06:31:17.107863 5327 tablet_bootstrap.cc:492] T ddcad719a8944ed39e43c11cd2015f38 P 25bfab18c92d4f6cb8727ef443a59867: Bootstrap starting.
I20260928 06:31:17.115931 5285 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.129:34211
I20260928 06:31:17.116544 5285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
I20260928 06:31:17.120445 5327 log.cc:824] T ddcad719a8944ed39e43c11cd2015f38 P 25bfab18c92d4f6cb8727ef443a59867: Log is configured to *not* fsync() on all Append() calls
I20260928 06:31:17.124792 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 5285
I20260928 06:31:17.124979 5434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.129:34211 every 8 connection(s)
I20260928 06:31:17.125109 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.130:41535
--local_ip_for_outbound_sockets=127.22.225.130
--tserver_master_addrs=127.22.225.190:36479
--webserver_port=43959
--webserver_interface=127.22.225.130
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab
--principal=kudu/127.22.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:37031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:31:17.127465 5327 tablet_bootstrap.cc:492] T ddcad719a8944ed39e43c11cd2015f38 P 25bfab18c92d4f6cb8727ef443a59867: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:31:17.127848 5327 tablet_bootstrap.cc:492] T ddcad719a8944ed39e43c11cd2015f38 P 25bfab18c92d4f6cb8727ef443a59867: Bootstrap complete.
I20260928 06:31:17.129233 5327 ts_tablet_manager.cc:1397] T ddcad719a8944ed39e43c11cd2015f38 P 25bfab18c92d4f6cb8727ef443a59867: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.010s
Sep 28 06:31:17 dist-test-slave-d59m krb5kdc[4666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577077, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.129@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:31:17.130833 5327 raft_consensus.cc:348] T ddcad719a8944ed39e43c11cd2015f38 P 25bfab18c92d4f6cb8727ef443a59867 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25bfab18c92d4f6cb8727ef443a59867" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34211 } } peers { permanent_uuid: "87f7f1fcfb1f4202973d15f5827e8d13" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 38549 } } peers { permanent_uuid: "cf5ca4308f864408828089cc6a5532af" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41535 } }
I20260928 06:31:17.131178 5327 raft_consensus.cc:729] T ddcad719a8944ed39e43c11cd2015f38 P 25bfab18c92d4f6cb8727ef443a59867 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25bfab18c92d4f6cb8727ef443a59867, State: Initialized, Role: FOLLOWER
I20260928 06:31:17.131362 5327 consensus_queue.cc:260] T ddcad719a8944ed39e43c11cd2015f38 P 25bfab18c92d4f6cb8727ef443a59867 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25bfab18c92d4f6cb8727ef443a59867" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34211 } } peers { permanent_uuid: "87f7f1fcfb1f4202973d15f5827e8d13" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 38549 } } peers { permanent_uuid: "cf5ca4308f864408828089cc6a5532af" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41535 } }
I20260928 06:31:17.131718 5327 ts_tablet_manager.cc:1428] T ddcad719a8944ed39e43c11cd2015f38 P 25bfab18c92d4f6cb8727ef443a59867: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260928 06:31:17.137171 5438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:17.117514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:47255 (local address 127.22.225.190:36479)
0928 06:31:17.121625 (+ 4111us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:17.121632 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:17.121669 (+ 37us) server_negotiation.cc:407] Connection header received
0928 06:31:17.121778 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:17.121783 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:17.121822 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:17.121970 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:17.122911 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:17.123796 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:17.127842 (+ 4046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:17.128561 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:17.128699 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:17.131900 (+ 3201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:17.131904 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:17.131904 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:17.131920 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:17.134330 (+ 2410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:17.134910 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:17.134914 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:17.134915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:17.135051 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:17.135384 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:17.135385 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:17.135385 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:17.135721 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:17.135809 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:17.136705 (+ 896us) server_negotiation.cc:299] Negotiation successful
0928 06:31:17.136861 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4058,"thread_start_us":82,"threads_started":1}
I20260928 06:31:17.138101 5437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:17.117445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36479 (local address 127.22.225.129:47255)
0928 06:31:17.120800 (+ 3355us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:17.120841 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:17.121324 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:17.122014 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:17.122017 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:17.122448 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:17.122757 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:17.122766 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:17.123939 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:17.123940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:17.127465 (+ 3525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:17.127468 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:17.128845 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:17.128846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:17.128999 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:17.129687 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:17.129698 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:17.131729 (+ 2031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:17.134443 (+ 2714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:17.134453 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:17.134456 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:17.134803 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:17.135133 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:17.135134 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:17.135135 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:17.135301 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:17.135846 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:17.135854 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:17.136027 (+ 173us) client_negotiation.cc:769] Sending connection context
0928 06:31:17.136144 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 06:31:17.136363 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3301,"thread_start_us":98,"threads_started":1}
I20260928 06:31:17.139302 5435 heartbeater.cc:347] Connected to a master server at 127.22.225.190:36479
I20260928 06:31:17.139402 5435 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:17.139619 5435 heartbeater.cc:511] Master 127.22.225.190:36479 requested a full tablet report, sending...
I20260928 06:31:17.140463 5247 ts_manager.cc:194] Registered new tserver with Master: 25bfab18c92d4f6cb8727ef443a59867 (127.22.225.129:34211)
I20260928 06:31:17.141822 5247 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.225.129@KRBTEST.COM'} at 127.22.225.129:47255
I20260928 06:31:17.143442 5435 heartbeater.cc:503] Master 127.22.225.190:36479 was elected leader, sending a full tablet report...
I20260928 06:31:17.148893 5442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:17.142365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.129:34211 (local address 127.0.0.1:33254)
0928 06:31:17.143863 (+ 1498us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:17.143907 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:17.144009 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:17.144388 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:17.144391 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:17.144394 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:17.144459 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:17.144461 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:17.145786 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:17.145788 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:17.147491 (+ 1703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:17.147495 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:17.148552 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:17.148553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:17.148663 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:17.148679 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 06:31:17.148750 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 06:31:17.148803 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1461,"thread_start_us":95,"threads_started":1}
I20260928 06:31:17.149147 5443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:17.142677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33254 (local address 127.22.225.129:34211)
0928 06:31:17.144169 (+ 1492us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:17.144174 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:17.144183 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:31:17.144224 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:17.144226 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:17.144242 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:17.144316 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:17.144546 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:17.145680 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:17.147602 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:17.148378 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:17.148455 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:17.148512 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:17.149021 (+ 509us) server_negotiation.cc:299] Negotiation successful
0928 06:31:17.149065 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1458,"thread_start_us":81,"threads_started":1}
I20260928 06:31:17.153584 5369 tablet_service.cc:1514] Processing DeleteTablet for tablet ddcad719a8944ed39e43c11cd2015f38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:31:16 UTC) from {username='kudu', principal='kudu/127.22.225.190@KRBTEST.COM'} at 127.0.0.1:33254
I20260928 06:31:17.154382 5444 tablet_replica.cc:330] stopping tablet replica
I20260928 06:31:17.154663 5444 raft_consensus.cc:2229] T ddcad719a8944ed39e43c11cd2015f38 P 25bfab18c92d4f6cb8727ef443a59867 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:31:17.154786 5444 raft_consensus.cc:2258] T ddcad719a8944ed39e43c11cd2015f38 P 25bfab18c92d4f6cb8727ef443a59867 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:31:17.155230 5444 ts_tablet_manager.cc:1905] T ddcad719a8944ed39e43c11cd2015f38 P 25bfab18c92d4f6cb8727ef443a59867: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:31:17.158043 5444 ts_tablet_manager.cc:1918] T ddcad719a8944ed39e43c11cd2015f38 P 25bfab18c92d4f6cb8727ef443a59867: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:31:17.158190 5444 log.cc:1196] T ddcad719a8944ed39e43c11cd2015f38 P 25bfab18c92d4f6cb8727ef443a59867: Deleting WAL directory at /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-0/wal/wals/ddcad719a8944ed39e43c11cd2015f38
I20260928 06:31:17.158623 5444 ts_tablet_manager.cc:1939] T ddcad719a8944ed39e43c11cd2015f38 P 25bfab18c92d4f6cb8727ef443a59867: Deleting consensus metadata
I20260928 06:31:17.159304 5233 catalog_manager.cc:4841] TS 25bfab18c92d4f6cb8727ef443a59867 (127.22.225.129:34211): tablet ddcad719a8944ed39e43c11cd2015f38 (table test-table [id=92c37192349e41e4b740a396b71bb5dd]) successfully deleted
W20260928 06:31:17.242098 5440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:17.242389 5440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:17.242436 5440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:17.243662 5440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:17.243731 5440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:17.243779 5440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.130
I20260928 06:31:17.245404 5440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:37031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab
--principal=kudu/127.22.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.130:41535
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.225.130
--webserver_port=43959
--tserver_master_addrs=127.22.225.190:36479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.130
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:31:17.245697 5440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:17.245926 5440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:17.251296 5450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:17.251276 5449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:17.251541 5440 server_base.cc:1034] running on GCE node
W20260928 06:31:17.251573 5452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:17.251828 5440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:17.252059 5440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:17.253245 5440 hybrid_clock.cc:648] HybridClock initialized: now 1790577077253230 us; error 34 us; skew 500 ppm
Sep 28 06:31:17 dist-test-slave-d59m krb5kdc[4666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577077, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:17.256274 5440 init.cc:377] Logged in from keytab as kudu/127.22.225.130@KRBTEST.COM (short username kudu)
I20260928 06:31:17.256829 5440 webserver.cc:460] Webserver started at http://127.22.225.130:43959/ using document root <none> and password file <none>
I20260928 06:31:17.257071 5440 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:17.257158 5440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:17.258563 5440 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:17.259352 5459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:17.259549 5440 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:17.259681 5440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-1/wal
uuid: "cf5ca4308f864408828089cc6a5532af"
format_stamp: "Formatted at 2026-09-28 06:31:16 on dist-test-slave-d59m"
server_key: "3af42ba6c7a3c7c037e2af04dc35f717"
server_key_iv: "30a0f8eeb009c9dce6d5b421f7b004f6"
server_key_version: "encryptionkey@0"
I20260928 06:31:17.260215 5440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:17.268852 5440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:17.271162 5440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:17.271317 5440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:17.271535 5440 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:31:17.272001 5466 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260928 06:31:17.273878 5466 ts_tablet_manager.cc:1852] T ddcad719a8944ed39e43c11cd2015f38 P cf5ca4308f864408828089cc6a5532af: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260928 06:31:17.274333 5466 ts_tablet_manager.cc:1905] T ddcad719a8944ed39e43c11cd2015f38 P cf5ca4308f864408828089cc6a5532af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:31:17.277278 5466 ts_tablet_manager.cc:1918] T ddcad719a8944ed39e43c11cd2015f38 P cf5ca4308f864408828089cc6a5532af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:31:17.277395 5466 log.cc:1196] T ddcad719a8944ed39e43c11cd2015f38 P cf5ca4308f864408828089cc6a5532af: Deleting WAL directory at /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-1/wal/wals/ddcad719a8944ed39e43c11cd2015f38
I20260928 06:31:17.277735 5466 ts_tablet_manager.cc:1939] T ddcad719a8944ed39e43c11cd2015f38 P cf5ca4308f864408828089cc6a5532af: Deleting consensus metadata
I20260928 06:31:17.278013 5440 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260928 06:31:17.278061 5440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260928 06:31:17.278107 5440 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:31:17.278137 5440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:17.286664 5440 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.130:41535
I20260928 06:31:17.286769 5573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.130:41535 every 8 connection(s)
I20260928 06:31:17.287129 5440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
Sep 28 06:31:17 dist-test-slave-d59m krb5kdc[4666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577077, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.130@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:31:17.295771 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 5440
I20260928 06:31:17.296044 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.131:38549
--local_ip_for_outbound_sockets=127.22.225.131
--tserver_master_addrs=127.22.225.190:36479
--webserver_port=39043
--webserver_interface=127.22.225.131
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab
--principal=kudu/127.22.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:37031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:31:17.298038 5438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:17.288006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:44929 (local address 127.22.225.190:36479)
0928 06:31:17.288077 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:17.288079 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:17.288283 (+ 204us) server_negotiation.cc:407] Connection header received
0928 06:31:17.288898 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:17.288900 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:17.288913 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:17.288995 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:17.289666 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:17.290478 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:17.291546 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:17.291913 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:17.291988 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:17.294597 (+ 2609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:17.294599 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:17.294599 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:17.294608 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:17.296214 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:17.296800 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:17.296802 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:17.296802 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:17.296869 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:17.297189 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:17.297190 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:17.297190 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:17.297341 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:17.297398 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:17.297802 (+ 404us) server_negotiation.cc:299] Negotiation successful
0928 06:31:17.297910 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 06:31:17.298175 5576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:17.287977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36479 (local address 127.22.225.130:44929)
0928 06:31:17.288147 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:17.288191 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:17.288790 (+ 599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:17.289075 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:17.289080 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:17.289345 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:17.289545 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:17.289549 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:17.290569 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:17.290569 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:17.291459 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:17.291462 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:17.292005 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:17.292006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:17.292122 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:17.292865 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:17.292875 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:17.294501 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:17.296315 (+ 1814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:17.296323 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:17.296326 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:17.296699 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:17.296944 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:17.296945 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:17.296945 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:17.297090 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:17.297437 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:17.297444 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:17.297618 (+ 174us) client_negotiation.cc:769] Sending connection context
0928 06:31:17.297733 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 06:31:17.297920 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":87,"threads_started":1}
I20260928 06:31:17.298883 5574 heartbeater.cc:347] Connected to a master server at 127.22.225.190:36479
I20260928 06:31:17.298995 5574 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:17.299185 5574 heartbeater.cc:511] Master 127.22.225.190:36479 requested a full tablet report, sending...
I20260928 06:31:17.299582 5247 ts_manager.cc:194] Registered new tserver with Master: cf5ca4308f864408828089cc6a5532af (127.22.225.130:41535)
I20260928 06:31:17.299969 5247 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.225.130@KRBTEST.COM'} at 127.22.225.130:44929
W20260928 06:31:17.395409 5577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:17.395640 5577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:17.395681 5577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:17.396939 5577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:17.396997 5577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:17.397042 5577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.131
I20260928 06:31:17.398518 5577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:37031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab
--principal=kudu/127.22.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.131:38549
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.225.131
--webserver_port=39043
--tserver_master_addrs=127.22.225.190:36479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.131
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:31:17.398779 5577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:17.399003 5577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:17.404155 5585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:17.404179 5582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:17.404250 5577 server_base.cc:1034] running on GCE node
W20260928 06:31:17.404353 5583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:17.404536 5577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:17.404848 5577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:17.406002 5577 hybrid_clock.cc:648] HybridClock initialized: now 1790577077405991 us; error 33 us; skew 500 ppm
Sep 28 06:31:17 dist-test-slave-d59m krb5kdc[4666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577077, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:17.408437 5577 init.cc:377] Logged in from keytab as kudu/127.22.225.131@KRBTEST.COM (short username kudu)
I20260928 06:31:17.408939 5577 webserver.cc:460] Webserver started at http://127.22.225.131:39043/ using document root <none> and password file <none>
I20260928 06:31:17.409178 5577 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:17.409266 5577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:17.410637 5577 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:17.411278 5592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:17.411459 5577 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:17.411563 5577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-2/wal
uuid: "87f7f1fcfb1f4202973d15f5827e8d13"
format_stamp: "Formatted at 2026-09-28 06:31:16 on dist-test-slave-d59m"
server_key: "0092f1137bc07d026e369fbaaa9094b9"
server_key_iv: "180de78533ad148654ac8275a18e1361"
server_key_version: "encryptionkey@0"
I20260928 06:31:17.411917 5577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:17.422204 5577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:17.424538 5577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:17.424753 5577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:17.425001 5577 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:31:17.425312 5577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:31:17.425364 5577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:17.425405 5577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:31:17.425444 5577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:17.431859 5577 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.131:38549
I20260928 06:31:17.431903 5705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.131:38549 every 8 connection(s)
I20260928 06:31:17.432276 5577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
I20260928 06:31:17.432432 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 5577
Sep 28 06:31:17 dist-test-slave-d59m krb5kdc[4666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577077, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.131@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:31:17.440807 5438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:17.433347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33970 (local address 127.22.225.190:36479)
0928 06:31:17.433403 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:17.433405 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:17.433417 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:31:17.433455 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:17.433456 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:17.433468 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:17.433526 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:17.434262 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:17.435060 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:17.436226 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:17.436505 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:17.436628 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:17.438075 (+ 1447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:17.438077 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:17.438077 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:17.438085 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:17.439657 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:17.440042 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:17.440043 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:17.440043 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:17.440104 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:17.440277 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:17.440277 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:17.440277 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:17.440387 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:17.440557 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:17.440614 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 06:31:17.440710 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 06:31:17.443899 5710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:17.433379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:50621 (local address 127.22.225.190:36479)
0928 06:31:17.433513 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:17.433515 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:17.433545 (+ 30us) server_negotiation.cc:407] Connection header received
0928 06:31:17.434088 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:17.434089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:17.434102 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:17.434163 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:17.435333 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:17.435936 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:17.437297 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:17.437507 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:17.437584 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:17.440167 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:17.440168 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:17.440168 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:17.440173 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:17.442585 (+ 2412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:17.442975 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:17.442977 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:17.442977 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:17.443046 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:17.443278 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:17.443279 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:17.443279 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:17.443410 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:17.443478 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:17.443720 (+ 242us) server_negotiation.cc:299] Negotiation successful
0928 06:31:17.443800 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":70,"threads_started":1}
I20260928 06:31:17.443988 5709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:17.433034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36479 (local address 127.22.225.131:50621)
0928 06:31:17.433408 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:17.433470 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:17.433980 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:17.434235 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:17.434239 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:17.434968 (+ 729us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:17.435249 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:17.435255 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:17.436045 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:17.436046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:17.437194 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:17.437197 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:17.437598 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:17.437600 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:17.437734 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:17.438393 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:17.438404 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:17.439907 (+ 1503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:17.442675 (+ 2768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:17.442681 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:17.442683 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:17.442894 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:17.443121 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:17.443122 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:17.443122 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:17.443220 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:17.443482 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:17.443486 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:17.443597 (+ 111us) client_negotiation.cc:769] Sending connection context
0928 06:31:17.443682 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 06:31:17.443797 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":87,"threads_started":1}
I20260928 06:31:17.444515 5706 heartbeater.cc:347] Connected to a master server at 127.22.225.190:36479
I20260928 06:31:17.444658 5706 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:17.444849 5706 heartbeater.cc:511] Master 127.22.225.190:36479 requested a full tablet report, sending...
I20260928 06:31:17.445215 5247 ts_manager.cc:194] Registered new tserver with Master: 87f7f1fcfb1f4202973d15f5827e8d13 (127.22.225.131:38549)
I20260928 06:31:17.445582 5247 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.225.131@KRBTEST.COM'} at 127.22.225.131:50621
I20260928 06:31:17.452389 23430 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:31:17.455263 5710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:17.452798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33976 (local address 127.22.225.190:36479)
0928 06:31:17.452843 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:17.452845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:17.452947 (+ 102us) server_negotiation.cc:407] Connection header received
0928 06:31:17.453006 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:17.453007 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:17.453038 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:17.453119 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:31:17.453267 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:17.453726 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:17.454476 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:17.454684 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:17.454758 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:17.454936 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:31:17.455031 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:31:17.455115 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:17.455161 (+ 46us) server_negotiation.cc:299] Negotiation successful
0928 06:31:17.455184 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:31:17.455505 5247 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:33976:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:31:17.457208 5247 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:31:17.459352 5369 tablet_service.cc:1467] Processing CreateTablet for tablet d4b959b495d944bcbc51d50c7f194871 (DEFAULT_TABLE table=test-table [id=39378ef1c3bd41b3b734618b7230a58c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:31:17.459558 5369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4b959b495d944bcbc51d50c7f194871. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:17.461410 5327 tablet_bootstrap.cc:492] T d4b959b495d944bcbc51d50c7f194871 P 25bfab18c92d4f6cb8727ef443a59867: Bootstrap starting.
I20260928 06:31:17.461886 5327 tablet_bootstrap.cc:654] T d4b959b495d944bcbc51d50c7f194871 P 25bfab18c92d4f6cb8727ef443a59867: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:17.463130 5712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:17.459176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.130:41535 (local address 127.0.0.1:57654)
0928 06:31:17.459372 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:17.459383 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:17.459433 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:17.459637 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:17.459638 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:17.459640 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:17.459701 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:17.459703 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:17.460726 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:17.460727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:17.461917 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:17.461920 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:17.462903 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:17.462905 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:17.462981 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:17.462985 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:31:17.463015 (+ 30us) client_negotiation.cc:240] Negotiation successful
0928 06:31:17.463052 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":75,"threads_started":1}
I20260928 06:31:17.463479 5713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:17.459301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57654 (local address 127.22.225.130:41535)
0928 06:31:17.459471 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:17.459476 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:17.459488 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:31:17.459530 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:17.459532 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:17.459551 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:17.459635 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:17.459768 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:17.460616 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:17.462026 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:17.462802 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:17.463237 (+ 435us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:17.463309 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:17.463337 (+ 28us) server_negotiation.cc:299] Negotiation successful
0928 06:31:17.463379 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":94,"threads_started":1}
I20260928 06:31:17.464004 5442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:17.459059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.131:38549 (local address 127.0.0.1:37108)
0928 06:31:17.459112 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:17.459124 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:17.459171 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:17.460315 (+ 1144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:17.460316 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:17.460318 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:17.460351 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:17.460353 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:17.461629 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:17.461630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:17.462723 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:17.462726 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:17.463749 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:17.463750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:17.463820 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:17.463824 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:31:17.463884 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 06:31:17.463927 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":13}
I20260928 06:31:17.464140 5327 tablet_bootstrap.cc:492] T d4b959b495d944bcbc51d50c7f194871 P 25bfab18c92d4f6cb8727ef443a59867: No bootstrap required, opened a new log
I20260928 06:31:17.464054 5508 tablet_service.cc:1467] Processing CreateTablet for tablet d4b959b495d944bcbc51d50c7f194871 (DEFAULT_TABLE table=test-table [id=39378ef1c3bd41b3b734618b7230a58c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:31:17.464198 5327 ts_tablet_manager.cc:1397] T d4b959b495d944bcbc51d50c7f194871 P 25bfab18c92d4f6cb8727ef443a59867: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.000s
I20260928 06:31:17.464293 5714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:17.459494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37108 (local address 127.22.225.131:38549)
0928 06:31:17.459846 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:17.459851 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:17.459860 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:31:17.459895 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:17.459898 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:17.459912 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:17.460300 (+ 388us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:17.460422 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:17.461495 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:17.462876 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:17.463654 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:17.463732 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:17.463783 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:17.464164 (+ 381us) server_negotiation.cc:299] Negotiation successful
0928 06:31:17.464214 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":101,"threads_started":1}
I20260928 06:31:17.464396 5508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4b959b495d944bcbc51d50c7f194871. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:17.464356 5327 raft_consensus.cc:348] T d4b959b495d944bcbc51d50c7f194871 P 25bfab18c92d4f6cb8727ef443a59867 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f7f1fcfb1f4202973d15f5827e8d13" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 38549 } } peers { permanent_uuid: "cf5ca4308f864408828089cc6a5532af" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41535 } } peers { permanent_uuid: "25bfab18c92d4f6cb8727ef443a59867" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34211 } }
I20260928 06:31:17.464434 5327 raft_consensus.cc:374] T d4b959b495d944bcbc51d50c7f194871 P 25bfab18c92d4f6cb8727ef443a59867 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:17.464480 5327 raft_consensus.cc:729] T d4b959b495d944bcbc51d50c7f194871 P 25bfab18c92d4f6cb8727ef443a59867 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25bfab18c92d4f6cb8727ef443a59867, State: Initialized, Role: FOLLOWER
I20260928 06:31:17.464529 5327 consensus_queue.cc:260] T d4b959b495d944bcbc51d50c7f194871 P 25bfab18c92d4f6cb8727ef443a59867 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f7f1fcfb1f4202973d15f5827e8d13" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 38549 } } peers { permanent_uuid: "cf5ca4308f864408828089cc6a5532af" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41535 } } peers { permanent_uuid: "25bfab18c92d4f6cb8727ef443a59867" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34211 } }
I20260928 06:31:17.464643 5327 ts_tablet_manager.cc:1428] T d4b959b495d944bcbc51d50c7f194871 P 25bfab18c92d4f6cb8727ef443a59867: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 06:31:17.464931 5640 tablet_service.cc:1467] Processing CreateTablet for tablet d4b959b495d944bcbc51d50c7f194871 (DEFAULT_TABLE table=test-table [id=39378ef1c3bd41b3b734618b7230a58c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:31:17.465238 5640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4b959b495d944bcbc51d50c7f194871. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:17.466475 5466 tablet_bootstrap.cc:492] T d4b959b495d944bcbc51d50c7f194871 P cf5ca4308f864408828089cc6a5532af: Bootstrap starting.
I20260928 06:31:17.467269 5466 tablet_bootstrap.cc:654] T d4b959b495d944bcbc51d50c7f194871 P cf5ca4308f864408828089cc6a5532af: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:17.467505 5466 log.cc:824] T d4b959b495d944bcbc51d50c7f194871 P cf5ca4308f864408828089cc6a5532af: Log is configured to *not* fsync() on all Append() calls
I20260928 06:31:17.468190 5466 tablet_bootstrap.cc:492] T d4b959b495d944bcbc51d50c7f194871 P cf5ca4308f864408828089cc6a5532af: No bootstrap required, opened a new log
I20260928 06:31:17.468230 5466 ts_tablet_manager.cc:1397] T d4b959b495d944bcbc51d50c7f194871 P cf5ca4308f864408828089cc6a5532af: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260928 06:31:17.468822 5717 tablet_bootstrap.cc:492] T d4b959b495d944bcbc51d50c7f194871 P 87f7f1fcfb1f4202973d15f5827e8d13: Bootstrap starting.
I20260928 06:31:17.469480 5717 tablet_bootstrap.cc:654] T d4b959b495d944bcbc51d50c7f194871 P 87f7f1fcfb1f4202973d15f5827e8d13: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:17.470031 5717 log.cc:824] T d4b959b495d944bcbc51d50c7f194871 P 87f7f1fcfb1f4202973d15f5827e8d13: Log is configured to *not* fsync() on all Append() calls
I20260928 06:31:17.470340 5466 raft_consensus.cc:348] T d4b959b495d944bcbc51d50c7f194871 P cf5ca4308f864408828089cc6a5532af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f7f1fcfb1f4202973d15f5827e8d13" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 38549 } } peers { permanent_uuid: "cf5ca4308f864408828089cc6a5532af" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41535 } } peers { permanent_uuid: "25bfab18c92d4f6cb8727ef443a59867" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34211 } }
I20260928 06:31:17.470505 5466 raft_consensus.cc:374] T d4b959b495d944bcbc51d50c7f194871 P cf5ca4308f864408828089cc6a5532af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:17.470548 5466 raft_consensus.cc:729] T d4b959b495d944bcbc51d50c7f194871 P cf5ca4308f864408828089cc6a5532af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf5ca4308f864408828089cc6a5532af, State: Initialized, Role: FOLLOWER
I20260928 06:31:17.470665 5466 consensus_queue.cc:260] T d4b959b495d944bcbc51d50c7f194871 P cf5ca4308f864408828089cc6a5532af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f7f1fcfb1f4202973d15f5827e8d13" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 38549 } } peers { permanent_uuid: "cf5ca4308f864408828089cc6a5532af" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41535 } } peers { permanent_uuid: "25bfab18c92d4f6cb8727ef443a59867" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34211 } }
I20260928 06:31:17.470938 5466 ts_tablet_manager.cc:1428] T d4b959b495d944bcbc51d50c7f194871 P cf5ca4308f864408828089cc6a5532af: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:31:17.470975 5574 heartbeater.cc:503] Master 127.22.225.190:36479 was elected leader, sending a full tablet report...
I20260928 06:31:17.472898 5717 tablet_bootstrap.cc:492] T d4b959b495d944bcbc51d50c7f194871 P 87f7f1fcfb1f4202973d15f5827e8d13: No bootstrap required, opened a new log
I20260928 06:31:17.472990 5717 ts_tablet_manager.cc:1397] T d4b959b495d944bcbc51d50c7f194871 P 87f7f1fcfb1f4202973d15f5827e8d13: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260928 06:31:17.474694 5717 raft_consensus.cc:348] T d4b959b495d944bcbc51d50c7f194871 P 87f7f1fcfb1f4202973d15f5827e8d13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f7f1fcfb1f4202973d15f5827e8d13" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 38549 } } peers { permanent_uuid: "cf5ca4308f864408828089cc6a5532af" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41535 } } peers { permanent_uuid: "25bfab18c92d4f6cb8727ef443a59867" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34211 } }
I20260928 06:31:17.474855 5717 raft_consensus.cc:374] T d4b959b495d944bcbc51d50c7f194871 P 87f7f1fcfb1f4202973d15f5827e8d13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:17.474902 5717 raft_consensus.cc:729] T d4b959b495d944bcbc51d50c7f194871 P 87f7f1fcfb1f4202973d15f5827e8d13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87f7f1fcfb1f4202973d15f5827e8d13, State: Initialized, Role: FOLLOWER
I20260928 06:31:17.475041 5717 consensus_queue.cc:260] T d4b959b495d944bcbc51d50c7f194871 P 87f7f1fcfb1f4202973d15f5827e8d13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f7f1fcfb1f4202973d15f5827e8d13" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 38549 } } peers { permanent_uuid: "cf5ca4308f864408828089cc6a5532af" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41535 } } peers { permanent_uuid: "25bfab18c92d4f6cb8727ef443a59867" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34211 } }
I20260928 06:31:17.475315 5717 ts_tablet_manager.cc:1428] T d4b959b495d944bcbc51d50c7f194871 P 87f7f1fcfb1f4202973d15f5827e8d13: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 06:31:17.475347 5706 heartbeater.cc:503] Master 127.22.225.190:36479 was elected leader, sending a full tablet report...
I20260928 06:31:17.490244 5441 raft_consensus.cc:482] T d4b959b495d944bcbc51d50c7f194871 P 25bfab18c92d4f6cb8727ef443a59867 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:31:17.490410 5441 raft_consensus.cc:504] T d4b959b495d944bcbc51d50c7f194871 P 25bfab18c92d4f6cb8727ef443a59867 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f7f1fcfb1f4202973d15f5827e8d13" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 38549 } } peers { permanent_uuid: "cf5ca4308f864408828089cc6a5532af" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41535 } } peers { permanent_uuid: "25bfab18c92d4f6cb8727ef443a59867" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34211 } }
I20260928 06:31:17.490787 5441 leader_election.cc:288] T d4b959b495d944bcbc51d50c7f194871 P 25bfab18c92d4f6cb8727ef443a59867 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87f7f1fcfb1f4202973d15f5827e8d13 (127.22.225.131:38549), cf5ca4308f864408828089cc6a5532af (127.22.225.130:41535)
I20260928 06:31:17.494553 5713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:17.491327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:38499 (local address 127.22.225.130:41535)
0928 06:31:17.491371 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:17.491373 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:17.491380 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:31:17.491405 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:17.491405 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:17.491418 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:17.491471 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:17.491642 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:17.492364 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:17.493579 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:17.494159 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:17.494243 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:17.494255 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:17.494428 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 06:31:17.494477 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:31:17.494575 5437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:17.490965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.131:38549 (local address 127.22.225.129:38569)
0928 06:31:17.491030 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:17.491043 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:17.491089 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:17.491273 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:17.491274 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:17.491276 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:17.491357 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:17.491359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:17.492320 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:17.492321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:17.493564 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:17.493567 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:17.494305 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:17.494306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:17.494380 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:17.494382 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:31:17.494447 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 06:31:17.494485 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 06:31:17.494820 5714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:17.491035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:38569 (local address 127.22.225.131:38549)
0928 06:31:17.491088 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:17.491090 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:17.491104 (+ 14us) server_negotiation.cc:407] Connection header received
0928 06:31:17.491139 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:17.491140 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:17.491150 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:17.491217 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:17.491433 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:17.492236 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:17.493642 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:17.494205 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:17.494265 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:17.494273 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:17.494677 (+ 404us) server_negotiation.cc:299] Negotiation successful
0928 06:31:17.494738 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 06:31:17.494591 5721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:17.491095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.130:41535 (local address 127.22.225.129:38499)
0928 06:31:17.491232 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:17.491242 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:17.491285 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:17.491476 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:17.491477 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:17.491479 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:17.491562 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:17.491564 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:17.492435 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:17.492436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:17.493490 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:17.493492 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:17.494272 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:17.494273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:17.494345 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:17.494350 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:31:17.494411 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 06:31:17.494451 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":10,"thread_start_us":91,"threads_started":1}
I20260928 06:31:17.495139 5660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4b959b495d944bcbc51d50c7f194871" candidate_uuid: "25bfab18c92d4f6cb8727ef443a59867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87f7f1fcfb1f4202973d15f5827e8d13" is_pre_election: true
I20260928 06:31:17.495313 5660 raft_consensus.cc:2454] T d4b959b495d944bcbc51d50c7f194871 P 87f7f1fcfb1f4202973d15f5827e8d13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25bfab18c92d4f6cb8727ef443a59867 in term 0.
I20260928 06:31:17.495249 5528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4b959b495d944bcbc51d50c7f194871" candidate_uuid: "25bfab18c92d4f6cb8727ef443a59867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf5ca4308f864408828089cc6a5532af" is_pre_election: true
I20260928 06:31:17.495421 5528 raft_consensus.cc:2454] T d4b959b495d944bcbc51d50c7f194871 P cf5ca4308f864408828089cc6a5532af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25bfab18c92d4f6cb8727ef443a59867 in term 0.
I20260928 06:31:17.495568 5324 leader_election.cc:302] T d4b959b495d944bcbc51d50c7f194871 P 25bfab18c92d4f6cb8727ef443a59867 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25bfab18c92d4f6cb8727ef443a59867, 87f7f1fcfb1f4202973d15f5827e8d13; no voters:
I20260928 06:31:17.495739 5441 raft_consensus.cc:2781] T d4b959b495d944bcbc51d50c7f194871 P 25bfab18c92d4f6cb8727ef443a59867 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:31:17.495791 5441 raft_consensus.cc:482] T d4b959b495d944bcbc51d50c7f194871 P 25bfab18c92d4f6cb8727ef443a59867 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:31:17.495812 5441 raft_consensus.cc:3037] T d4b959b495d944bcbc51d50c7f194871 P 25bfab18c92d4f6cb8727ef443a59867 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:17.496775 5441 raft_consensus.cc:504] T d4b959b495d944bcbc51d50c7f194871 P 25bfab18c92d4f6cb8727ef443a59867 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f7f1fcfb1f4202973d15f5827e8d13" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 38549 } } peers { permanent_uuid: "cf5ca4308f864408828089cc6a5532af" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41535 } } peers { permanent_uuid: "25bfab18c92d4f6cb8727ef443a59867" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34211 } }
I20260928 06:31:17.496912 5441 leader_election.cc:288] T d4b959b495d944bcbc51d50c7f194871 P 25bfab18c92d4f6cb8727ef443a59867 [CANDIDATE]: Term 1 election: Requested vote from peers 87f7f1fcfb1f4202973d15f5827e8d13 (127.22.225.131:38549), cf5ca4308f864408828089cc6a5532af (127.22.225.130:41535)
I20260928 06:31:17.497112 5660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4b959b495d944bcbc51d50c7f194871" candidate_uuid: "25bfab18c92d4f6cb8727ef443a59867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87f7f1fcfb1f4202973d15f5827e8d13"
I20260928 06:31:17.497097 5528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4b959b495d944bcbc51d50c7f194871" candidate_uuid: "25bfab18c92d4f6cb8727ef443a59867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf5ca4308f864408828089cc6a5532af"
I20260928 06:31:17.497188 5528 raft_consensus.cc:3037] T d4b959b495d944bcbc51d50c7f194871 P cf5ca4308f864408828089cc6a5532af [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:17.497188 5660 raft_consensus.cc:3037] T d4b959b495d944bcbc51d50c7f194871 P 87f7f1fcfb1f4202973d15f5827e8d13 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:17.498108 5528 raft_consensus.cc:2454] T d4b959b495d944bcbc51d50c7f194871 P cf5ca4308f864408828089cc6a5532af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25bfab18c92d4f6cb8727ef443a59867 in term 1.
I20260928 06:31:17.498108 5660 raft_consensus.cc:2454] T d4b959b495d944bcbc51d50c7f194871 P 87f7f1fcfb1f4202973d15f5827e8d13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25bfab18c92d4f6cb8727ef443a59867 in term 1.
I20260928 06:31:17.498296 5324 leader_election.cc:302] T d4b959b495d944bcbc51d50c7f194871 P 25bfab18c92d4f6cb8727ef443a59867 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25bfab18c92d4f6cb8727ef443a59867, 87f7f1fcfb1f4202973d15f5827e8d13; no voters:
I20260928 06:31:17.498404 5441 raft_consensus.cc:2781] T d4b959b495d944bcbc51d50c7f194871 P 25bfab18c92d4f6cb8727ef443a59867 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:31:17.498591 5441 raft_consensus.cc:686] T d4b959b495d944bcbc51d50c7f194871 P 25bfab18c92d4f6cb8727ef443a59867 [term 1 LEADER]: Becoming Leader. State: Replica: 25bfab18c92d4f6cb8727ef443a59867, State: Running, Role: LEADER
I20260928 06:31:17.498697 5441 consensus_queue.cc:237] T d4b959b495d944bcbc51d50c7f194871 P 25bfab18c92d4f6cb8727ef443a59867 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f7f1fcfb1f4202973d15f5827e8d13" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 38549 } } peers { permanent_uuid: "cf5ca4308f864408828089cc6a5532af" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41535 } } peers { permanent_uuid: "25bfab18c92d4f6cb8727ef443a59867" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34211 } }
I20260928 06:31:17.499441 5245 catalog_manager.cc:5495] T d4b959b495d944bcbc51d50c7f194871 P 25bfab18c92d4f6cb8727ef443a59867 reported cstate change: term changed from 0 to 1, leader changed from <none> to 25bfab18c92d4f6cb8727ef443a59867 (127.22.225.129). New cstate: current_term: 1 leader_uuid: "25bfab18c92d4f6cb8727ef443a59867" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f7f1fcfb1f4202973d15f5827e8d13" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 38549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf5ca4308f864408828089cc6a5532af" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25bfab18c92d4f6cb8727ef443a59867" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34211 } health_report { overall_health: HEALTHY } } }
I20260928 06:31:17.513633 5443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:17.510862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33256 (local address 127.22.225.129:34211)
0928 06:31:17.510927 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:17.510929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:17.510940 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:31:17.510991 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:17.510993 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:17.511021 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:17.511118 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:31:17.511212 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:17.511978 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:17.512721 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:17.513055 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:17.513137 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:17.513244 (+ 107us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:31:17.513398 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:31:17.513469 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:17.513499 (+ 30us) server_negotiation.cc:299] Negotiation successful
0928 06:31:17.513526 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 06:31:17.515233 5660 raft_consensus.cc:1261] T d4b959b495d944bcbc51d50c7f194871 P 87f7f1fcfb1f4202973d15f5827e8d13 [term 1 FOLLOWER]: Refusing update from remote peer 25bfab18c92d4f6cb8727ef443a59867: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:31:17.515282 5528 raft_consensus.cc:1261] T d4b959b495d944bcbc51d50c7f194871 P cf5ca4308f864408828089cc6a5532af [term 1 FOLLOWER]: Refusing update from remote peer 25bfab18c92d4f6cb8727ef443a59867: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:31:17.515627 5441 consensus_queue.cc:1035] T d4b959b495d944bcbc51d50c7f194871 P 25bfab18c92d4f6cb8727ef443a59867 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87f7f1fcfb1f4202973d15f5827e8d13" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 38549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:31:17.515796 5722 consensus_queue.cc:1035] T d4b959b495d944bcbc51d50c7f194871 P 25bfab18c92d4f6cb8727ef443a59867 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf5ca4308f864408828089cc6a5532af" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:31:17.520618 5726 mvcc.cc:205] Tried to move back new op lower bound from 7334203709499621376 to 7334203709435260928. Current Snapshot: MvccSnapshot[applied={T|T < 7334203709499621376}]
I20260928 06:31:17.520640 5725 mvcc.cc:205] Tried to move back new op lower bound from 7334203709499621376 to 7334203709435260928. Current Snapshot: MvccSnapshot[applied={T|T < 7334203709499621376}]
I20260928 06:31:17.521097 5246 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:33976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:31:17.521204 5246 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:33976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:31:17.521957 5246 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fc5064433d164e12b54242dc5cb206fd: Sending DeleteTablet for 3 replicas of tablet d4b959b495d944bcbc51d50c7f194871
I20260928 06:31:17.522295 5369 tablet_service.cc:1514] Processing DeleteTablet for tablet d4b959b495d944bcbc51d50c7f194871 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:31:17 UTC) from {username='kudu', principal='kudu/127.22.225.190@KRBTEST.COM'} at 127.0.0.1:33254
I20260928 06:31:17.522339 5640 tablet_service.cc:1514] Processing DeleteTablet for tablet d4b959b495d944bcbc51d50c7f194871 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:31:17 UTC) from {username='kudu', principal='kudu/127.22.225.190@KRBTEST.COM'} at 127.0.0.1:37108
I20260928 06:31:17.522393 5444 tablet_replica.cc:330] T d4b959b495d944bcbc51d50c7f194871 P 25bfab18c92d4f6cb8727ef443a59867: stopping tablet replica
I20260928 06:31:17.522441 5444 raft_consensus.cc:2229] T d4b959b495d944bcbc51d50c7f194871 P 25bfab18c92d4f6cb8727ef443a59867 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:31:17.522414 5508 tablet_service.cc:1514] Processing DeleteTablet for tablet d4b959b495d944bcbc51d50c7f194871 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:31:17 UTC) from {username='kudu', principal='kudu/127.22.225.190@KRBTEST.COM'} at 127.0.0.1:57654
I20260928 06:31:17.522527 5444 raft_consensus.cc:2258] T d4b959b495d944bcbc51d50c7f194871 P 25bfab18c92d4f6cb8727ef443a59867 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:31:17.522631 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 5285
W20260928 06:31:17.528565 5233 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.225.129:34211 (error 108)
I20260928 06:31:17.528960 5736 tablet_replica.cc:330] T d4b959b495d944bcbc51d50c7f194871 P cf5ca4308f864408828089cc6a5532af: stopping tablet replica
I20260928 06:31:17.529100 5736 raft_consensus.cc:2229] T d4b959b495d944bcbc51d50c7f194871 P cf5ca4308f864408828089cc6a5532af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:31:17.529143 5442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:17.528993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.129:34211 (local address 127.0.0.1:33268)
0928 06:31:17.529057 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:17.529106 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.129:34211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 06:31:17.529255 5736 raft_consensus.cc:2258] T d4b959b495d944bcbc51d50c7f194871 P cf5ca4308f864408828089cc6a5532af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:31:17.529275 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 5440
I20260928 06:31:17.529446 5735 tablet_replica.cc:330] T d4b959b495d944bcbc51d50c7f194871 P 87f7f1fcfb1f4202973d15f5827e8d13: stopping tablet replica
I20260928 06:31:17.529554 5735 raft_consensus.cc:2229] T d4b959b495d944bcbc51d50c7f194871 P 87f7f1fcfb1f4202973d15f5827e8d13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:31:17.529654 5735 raft_consensus.cc:2258] T d4b959b495d944bcbc51d50c7f194871 P 87f7f1fcfb1f4202973d15f5827e8d13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:31:17.530586 5735 ts_tablet_manager.cc:1905] T d4b959b495d944bcbc51d50c7f194871 P 87f7f1fcfb1f4202973d15f5827e8d13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:31:17.532747 5735 ts_tablet_manager.cc:1918] T d4b959b495d944bcbc51d50c7f194871 P 87f7f1fcfb1f4202973d15f5827e8d13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:31:17.532829 5735 log.cc:1196] T d4b959b495d944bcbc51d50c7f194871 P 87f7f1fcfb1f4202973d15f5827e8d13: Deleting WAL directory at /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-2/wal/wals/d4b959b495d944bcbc51d50c7f194871
I20260928 06:31:17.533115 5735 ts_tablet_manager.cc:1939] T d4b959b495d944bcbc51d50c7f194871 P 87f7f1fcfb1f4202973d15f5827e8d13: Deleting consensus metadata
W20260928 06:31:17.532630 5233 catalog_manager.cc:4584] TS 25bfab18c92d4f6cb8727ef443a59867 (127.22.225.129:34211): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d4b959b495d944bcbc51d50c7f194871: Network error: Client connection negotiation failed: client connection to 127.22.225.129:34211: connect: Connection refused (error 111)
I20260928 06:31:17.533583 5234 catalog_manager.cc:4841] TS 87f7f1fcfb1f4202973d15f5827e8d13 (127.22.225.131:38549): tablet d4b959b495d944bcbc51d50c7f194871 (table test-table [id=39378ef1c3bd41b3b734618b7230a58c]) successfully deleted
I20260928 06:31:17.537355 5442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:17.537231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.130:41535 (local address 127.0.0.1:57658)
0928 06:31:17.537289 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:17.537318 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.130:41535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 06:31:17.537469 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 5577
I20260928 06:31:17.542583 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 5214
I20260928 06:31:17.550616 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:36479
--webserver_interface=127.22.225.190
--webserver_port=41111
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab
--principal=kudu/127.22.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:37031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.225.190:36479
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:31:17.638700 5738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:17.638911 5738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:17.638933 5738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:17.640089 5738 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:31:17.640133 5738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:17.640148 5738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:17.640161 5738 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:31:17.640172 5738 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:31:17.641690 5738 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:37031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.225.190:36479
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab
--principal=kudu/127.22.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:36479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.225.190
--webserver_port=41111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:31:17.641939 5738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:17.642182 5738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:17.647665 5744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:17.647675 5743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:17.647851 5738 server_base.cc:1034] running on GCE node
W20260928 06:31:17.647675 5746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:17.648216 5738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:17.648542 5738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:17.649730 5738 hybrid_clock.cc:648] HybridClock initialized: now 1790577077649709 us; error 39 us; skew 500 ppm
Sep 28 06:31:17 dist-test-slave-d59m krb5kdc[4666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577077, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:17.652427 5738 init.cc:377] Logged in from keytab as kudu/127.22.225.190@KRBTEST.COM (short username kudu)
I20260928 06:31:17.652926 5738 webserver.cc:460] Webserver started at http://127.22.225.190:41111/ using document root <none> and password file <none>
I20260928 06:31:17.653179 5738 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:17.653246 5738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:17.654616 5738 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:31:17.655328 5753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:17.655507 5738 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:31:17.655627 5738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/master-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/master-0/wal
uuid: "fc5064433d164e12b54242dc5cb206fd"
format_stamp: "Formatted at 2026-09-28 06:31:16 on dist-test-slave-d59m"
server_key: "8ff09d423c9b79596a1c63fe6d58dae1"
server_key_iv: "062ab7b58b2114e58aadc43f1f66401b"
server_key_version: "encryptionkey@0"
I20260928 06:31:17.655982 5738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:17.686790 5738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:17.689247 5738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:17.689414 5738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:17.693969 5738 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.190:36479
I20260928 06:31:17.694000 5805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.190:36479 every 8 connection(s)
I20260928 06:31:17.694388 5738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
I20260928 06:31:17.696419 5806 sys_catalog.cc:263] Verifying existing consensus state
I20260928 06:31:17.696419 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 5738
I20260928 06:31:17.696794 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.129:34211
--local_ip_for_outbound_sockets=127.22.225.129
--tserver_master_addrs=127.22.225.190:36479
--webserver_port=45539
--webserver_interface=127.22.225.129
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab
--principal=kudu/127.22.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:37031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:31:17.697265 5806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc5064433d164e12b54242dc5cb206fd: Bootstrap starting.
I20260928 06:31:17.708163 5806 log.cc:824] T 00000000000000000000000000000000 P fc5064433d164e12b54242dc5cb206fd: Log is configured to *not* fsync() on all Append() calls
I20260928 06:31:17.711079 5806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc5064433d164e12b54242dc5cb206fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:31:17.711418 5806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc5064433d164e12b54242dc5cb206fd: Bootstrap complete.
I20260928 06:31:17.714432 5806 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc5064433d164e12b54242dc5cb206fd [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc5064433d164e12b54242dc5cb206fd" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36479 } }
I20260928 06:31:17.714717 5806 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc5064433d164e12b54242dc5cb206fd [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc5064433d164e12b54242dc5cb206fd, State: Initialized, Role: FOLLOWER
I20260928 06:31:17.714884 5806 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc5064433d164e12b54242dc5cb206fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc5064433d164e12b54242dc5cb206fd" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36479 } }
I20260928 06:31:17.714949 5806 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc5064433d164e12b54242dc5cb206fd [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:31:17.714986 5806 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc5064433d164e12b54242dc5cb206fd [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:31:17.715034 5806 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc5064433d164e12b54242dc5cb206fd [term 2 FOLLOWER]: Advancing to term 3
I20260928 06:31:17.716300 5806 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc5064433d164e12b54242dc5cb206fd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc5064433d164e12b54242dc5cb206fd" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36479 } }
I20260928 06:31:17.716421 5806 leader_election.cc:302] T 00000000000000000000000000000000 P fc5064433d164e12b54242dc5cb206fd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc5064433d164e12b54242dc5cb206fd; no voters:
I20260928 06:31:17.716619 5806 leader_election.cc:288] T 00000000000000000000000000000000 P fc5064433d164e12b54242dc5cb206fd [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 06:31:17.716747 5810 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc5064433d164e12b54242dc5cb206fd [term 3 FOLLOWER]: Leader election won for term 3
I20260928 06:31:17.716919 5810 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc5064433d164e12b54242dc5cb206fd [term 3 LEADER]: Becoming Leader. State: Replica: fc5064433d164e12b54242dc5cb206fd, State: Running, Role: LEADER
I20260928 06:31:17.716962 5806 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc5064433d164e12b54242dc5cb206fd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:31:17.717120 5810 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc5064433d164e12b54242dc5cb206fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc5064433d164e12b54242dc5cb206fd" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36479 } }
I20260928 06:31:17.717804 5810 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc5064433d164e12b54242dc5cb206fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc5064433d164e12b54242dc5cb206fd. Latest consensus state: current_term: 3 leader_uuid: "fc5064433d164e12b54242dc5cb206fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc5064433d164e12b54242dc5cb206fd" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36479 } } }
I20260928 06:31:17.717957 5810 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc5064433d164e12b54242dc5cb206fd [sys.catalog]: This master's current role is: LEADER
I20260928 06:31:17.718034 5811 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc5064433d164e12b54242dc5cb206fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "fc5064433d164e12b54242dc5cb206fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc5064433d164e12b54242dc5cb206fd" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36479 } } }
I20260928 06:31:17.718140 5811 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc5064433d164e12b54242dc5cb206fd [sys.catalog]: This master's current role is: LEADER
I20260928 06:31:17.718588 5822 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:31:17.719410 5822 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:31:17.719492 5822 catalog_manager.cc:1259] Loaded cluster ID: 48889f2c3c5e41d0b73452627b7870a1
I20260928 06:31:17.719514 5822 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 06:31:17.719810 5822 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f56377259ad google::LogMessage::Fail() at ??:0
@ 0x7f563772888a google::LogMessage::SendToLog() at ??:0
@ 0x7f56377254de google::LogMessage::Flush() at ??:0
@ 0x7f5637726a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f563f6460f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f563f64f8b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f56395f30d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f56395e8954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f563a44c6db start_thread at ??:0
@ 0x7f563622361f clone at ??:0
W20260928 06:31:17.815382 5808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:17.815662 5808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:17.815701 5808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:17.817469 5808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:17.817536 5808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:17.817581 5808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.129
I20260928 06:31:17.819836 5808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:37031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab
--principal=kudu/127.22.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.129:34211
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.225.129
--webserver_port=45539
--tserver_master_addrs=127.22.225.190:36479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.129
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:31:17.820098 5808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:17.820367 5808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:17.826355 5843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:17.826499 5808 server_base.cc:1034] running on GCE node
W20260928 06:31:17.826607 5845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:17.826622 5842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:17.826869 5808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:17.827172 5808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:17.828352 5808 hybrid_clock.cc:648] HybridClock initialized: now 1790577077828333 us; error 34 us; skew 500 ppm
Sep 28 06:31:17 dist-test-slave-d59m krb5kdc[4666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577077, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:17.831620 5808 init.cc:377] Logged in from keytab as kudu/127.22.225.129@KRBTEST.COM (short username kudu)
I20260928 06:31:17.832226 5808 webserver.cc:460] Webserver started at http://127.22.225.129:45539/ using document root <none> and password file <none>
I20260928 06:31:17.832515 5808 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:17.832617 5808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:17.834467 5808 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:17.835307 5852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:17.835497 5808 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:17.835592 5808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-0/wal
uuid: "25bfab18c92d4f6cb8727ef443a59867"
format_stamp: "Formatted at 2026-09-28 06:31:16 on dist-test-slave-d59m"
server_key: "3c19ffd404f956b5dcc1ca39f58ab04f"
server_key_iv: "4e0b9e4fdda099feea9abffca4d91885"
server_key_version: "encryptionkey@0"
I20260928 06:31:17.836041 5808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:17.859619 5808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:17.862462 5808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:17.862634 5808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:17.862917 5808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:31:17.863490 5859 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 06:31:17.865389 5808 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 06:31:17.865461 5808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 06:31:17.865499 5808 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 06:31:17.866354 5808 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 06:31:17.866402 5808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 06:31:17.866564 5859 tablet_bootstrap.cc:492] T d4b959b495d944bcbc51d50c7f194871 P 25bfab18c92d4f6cb8727ef443a59867: Bootstrap starting.
I20260928 06:31:17.874210 5808 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.129:34211
I20260928 06:31:17.874706 5808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
I20260928 06:31:17.875243 5966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.129:34211 every 8 connection(s)
I20260928 06:31:17.877992 5859 log.cc:824] T d4b959b495d944bcbc51d50c7f194871 P 25bfab18c92d4f6cb8727ef443a59867: Log is configured to *not* fsync() on all Append() calls
I20260928 06:31:17.880617 5859 tablet_bootstrap.cc:492] T d4b959b495d944bcbc51d50c7f194871 P 25bfab18c92d4f6cb8727ef443a59867: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:31:17.880919 5859 tablet_bootstrap.cc:492] T d4b959b495d944bcbc51d50c7f194871 P 25bfab18c92d4f6cb8727ef443a59867: Bootstrap complete.
I20260928 06:31:17.882166 5859 ts_tablet_manager.cc:1397] T d4b959b495d944bcbc51d50c7f194871 P 25bfab18c92d4f6cb8727ef443a59867: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.003s
I20260928 06:31:17.883460 5859 raft_consensus.cc:348] T d4b959b495d944bcbc51d50c7f194871 P 25bfab18c92d4f6cb8727ef443a59867 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f7f1fcfb1f4202973d15f5827e8d13" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 38549 } } peers { permanent_uuid: "cf5ca4308f864408828089cc6a5532af" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41535 } } peers { permanent_uuid: "25bfab18c92d4f6cb8727ef443a59867" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34211 } }
I20260928 06:31:17.883754 5859 raft_consensus.cc:729] T d4b959b495d944bcbc51d50c7f194871 P 25bfab18c92d4f6cb8727ef443a59867 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25bfab18c92d4f6cb8727ef443a59867, State: Initialized, Role: FOLLOWER
I20260928 06:31:17.883886 5859 consensus_queue.cc:260] T d4b959b495d944bcbc51d50c7f194871 P 25bfab18c92d4f6cb8727ef443a59867 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f7f1fcfb1f4202973d15f5827e8d13" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 38549 } } peers { permanent_uuid: "cf5ca4308f864408828089cc6a5532af" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41535 } } peers { permanent_uuid: "25bfab18c92d4f6cb8727ef443a59867" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34211 } }
I20260928 06:31:17.884145 5859 ts_tablet_manager.cc:1428] T d4b959b495d944bcbc51d50c7f194871 P 25bfab18c92d4f6cb8727ef443a59867: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 06:31:17.884227 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 5808
I20260928 06:31:17.884516 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.130:41535
--local_ip_for_outbound_sockets=127.22.225.130
--tserver_master_addrs=127.22.225.190:36479
--webserver_port=43959
--webserver_interface=127.22.225.130
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab
--principal=kudu/127.22.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:37031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:31:17.973583 5969 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:31:17.875802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36479 (local address 127.22.225.129:53597)
0928 06:31:17.876245 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:17.876298 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:17.877038 (+ 740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:17.973359 (+ 96321us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:36479: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":101,"threads_started":1}
W20260928 06:31:17.973892 5967 heartbeater.cc:644] Failed to heartbeat to 127.22.225.190:36479 (0 consecutive failures): Network error: Failed to ping master at 127.22.225.190:36479: Client connection negotiation failed: client connection to 127.22.225.190:36479: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 06:31:17.974259 5969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:17.974102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36479 (local address 127.22.225.129:48597)
0928 06:31:17.974165 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:17.974228 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:36479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 06:31:17.974694 5969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:17.974580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36479 (local address 127.22.225.129:48853)
0928 06:31:17.974635 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:17.974667 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:36479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260928 06:31:17.974858 5967 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 06:31:17.996155 5972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:17.996416 5972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:17.996470 5972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:17.997695 5972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:17.997764 5972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:17.997810 5972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.130
I20260928 06:31:17.999286 5972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:37031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab
--principal=kudu/127.22.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.130:41535
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.225.130
--webserver_port=43959
--tserver_master_addrs=127.22.225.190:36479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.130
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:31:17.999562 5972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:17.999801 5972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:18.005107 5977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:18.005112 5978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:18.005103 5980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:18.005316 5972 server_base.cc:1034] running on GCE node
I20260928 06:31:18.005664 5972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:18.005945 5972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:18.007110 5972 hybrid_clock.cc:648] HybridClock initialized: now 1790577078007098 us; error 31 us; skew 500 ppm
Sep 28 06:31:18 dist-test-slave-d59m krb5kdc[4666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577078, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:18.009855 5972 init.cc:377] Logged in from keytab as kudu/127.22.225.130@KRBTEST.COM (short username kudu)
I20260928 06:31:18.010370 5972 webserver.cc:460] Webserver started at http://127.22.225.130:43959/ using document root <none> and password file <none>
I20260928 06:31:18.010618 5972 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:18.010666 5972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:18.012024 5972 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:18.012741 5987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:18.012895 5972 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:31:18.012990 5972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-1/wal
uuid: "cf5ca4308f864408828089cc6a5532af"
format_stamp: "Formatted at 2026-09-28 06:31:16 on dist-test-slave-d59m"
server_key: "3af42ba6c7a3c7c037e2af04dc35f717"
server_key_iv: "30a0f8eeb009c9dce6d5b421f7b004f6"
server_key_version: "encryptionkey@0"
I20260928 06:31:18.013341 5972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:18.028545 5972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:18.030885 5972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:18.031045 5972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:18.031314 5972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:31:18.031781 5994 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 06:31:18.033916 5972 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 06:31:18.033980 5972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 06:31:18.034026 5972 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 06:31:18.034693 5972 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 06:31:18.034746 5972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 06:31:18.034828 5994 tablet_bootstrap.cc:492] T d4b959b495d944bcbc51d50c7f194871 P cf5ca4308f864408828089cc6a5532af: Bootstrap starting.
I20260928 06:31:18.042245 5972 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.130:41535
I20260928 06:31:18.042641 5972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
I20260928 06:31:18.043057 6101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.130:41535 every 8 connection(s)
I20260928 06:31:18.044935 6104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:18.043344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36479 (local address 127.22.225.130:52753)
0928 06:31:18.044711 (+ 1367us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:18.044796 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:36479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1321,"thread_start_us":100,"threads_started":1}
W20260928 06:31:18.045231 6102 heartbeater.cc:644] Failed to heartbeat to 127.22.225.190:36479 (0 consecutive failures): Network error: Failed to ping master at 127.22.225.190:36479: Client connection negotiation failed: client connection to 127.22.225.190:36479: connect: Connection refused (error 111)
I20260928 06:31:18.045550 6104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:18.045455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36479 (local address 127.22.225.130:36517)
0928 06:31:18.045503 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:18.045533 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:36479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 06:31:18.045900 6104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:18.045819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36479 (local address 127.22.225.130:38401)
0928 06:31:18.045858 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:18.045880 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:36479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260928 06:31:18.046059 6102 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 06:31:18.046527 5994 log.cc:824] T d4b959b495d944bcbc51d50c7f194871 P cf5ca4308f864408828089cc6a5532af: Log is configured to *not* fsync() on all Append() calls
I20260928 06:31:18.049280 5994 tablet_bootstrap.cc:492] T d4b959b495d944bcbc51d50c7f194871 P cf5ca4308f864408828089cc6a5532af: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:31:18.049597 5994 tablet_bootstrap.cc:492] T d4b959b495d944bcbc51d50c7f194871 P cf5ca4308f864408828089cc6a5532af: Bootstrap complete.
I20260928 06:31:18.050843 5994 ts_tablet_manager.cc:1397] T d4b959b495d944bcbc51d50c7f194871 P cf5ca4308f864408828089cc6a5532af: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.009s
I20260928 06:31:18.050865 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 5972
I20260928 06:31:18.051100 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.131:38549
--local_ip_for_outbound_sockets=127.22.225.131
--tserver_master_addrs=127.22.225.190:36479
--webserver_port=39043
--webserver_interface=127.22.225.131
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab
--principal=kudu/127.22.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:37031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:31:18.052484 5994 raft_consensus.cc:348] T d4b959b495d944bcbc51d50c7f194871 P cf5ca4308f864408828089cc6a5532af [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f7f1fcfb1f4202973d15f5827e8d13" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 38549 } } peers { permanent_uuid: "cf5ca4308f864408828089cc6a5532af" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41535 } } peers { permanent_uuid: "25bfab18c92d4f6cb8727ef443a59867" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34211 } }
I20260928 06:31:18.052917 5994 raft_consensus.cc:729] T d4b959b495d944bcbc51d50c7f194871 P cf5ca4308f864408828089cc6a5532af [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf5ca4308f864408828089cc6a5532af, State: Initialized, Role: FOLLOWER
I20260928 06:31:18.053112 5994 consensus_queue.cc:260] T d4b959b495d944bcbc51d50c7f194871 P cf5ca4308f864408828089cc6a5532af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f7f1fcfb1f4202973d15f5827e8d13" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 38549 } } peers { permanent_uuid: "cf5ca4308f864408828089cc6a5532af" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41535 } } peers { permanent_uuid: "25bfab18c92d4f6cb8727ef443a59867" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34211 } }
I20260928 06:31:18.053372 5994 ts_tablet_manager.cc:1428] T d4b959b495d944bcbc51d50c7f194871 P cf5ca4308f864408828089cc6a5532af: Time spent starting tablet: real 0.002s user 0.000s sys 0.005s
I20260928 06:31:18.053653 6104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:18.053543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36479 (local address 127.22.225.130:34101)
0928 06:31:18.053590 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:18.053622 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:36479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 06:31:18.057075 6104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:18.056968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36479 (local address 127.22.225.130:37893)
0928 06:31:18.057015 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:18.057043 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:36479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 06:31:18.151875 6106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:18.152091 6106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:18.152115 6106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:18.153434 6106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:18.153479 6106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:18.153507 6106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.131
I20260928 06:31:18.155042 6106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:37031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab
--principal=kudu/127.22.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.131:38549
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.225.131
--webserver_port=39043
--tserver_master_addrs=127.22.225.190:36479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6106
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.131
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:31:18.155279 6106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:18.155509 6106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:18.160501 6115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:18.160472 6113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:18.160483 6112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:18.160832 6106 server_base.cc:1034] running on GCE node
I20260928 06:31:18.161031 6106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:18.161276 6106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:18.162463 6106 hybrid_clock.cc:648] HybridClock initialized: now 1790577078162448 us; error 33 us; skew 500 ppm
Sep 28 06:31:18 dist-test-slave-d59m krb5kdc[4666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577078, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:18.165045 6106 init.cc:377] Logged in from keytab as kudu/127.22.225.131@KRBTEST.COM (short username kudu)
I20260928 06:31:18.165524 6106 webserver.cc:460] Webserver started at http://127.22.225.131:39043/ using document root <none> and password file <none>
I20260928 06:31:18.165751 6106 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:18.165820 6106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:18.167205 6106 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:18.167862 6122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:18.168044 6106 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:31:18.168156 6106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-2/wal
uuid: "87f7f1fcfb1f4202973d15f5827e8d13"
format_stamp: "Formatted at 2026-09-28 06:31:16 on dist-test-slave-d59m"
server_key: "0092f1137bc07d026e369fbaaa9094b9"
server_key_iv: "180de78533ad148654ac8275a18e1361"
server_key_version: "encryptionkey@0"
I20260928 06:31:18.168520 6106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:18.182934 6106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:18.185496 6106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:18.185690 6106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:18.186013 6106 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:31:18.186411 6106 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:31:18.186460 6106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:18.186502 6106 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:31:18.186525 6106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:18.193913 6106 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.131:38549
I20260928 06:31:18.194020 6235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.131:38549 every 8 connection(s)
I20260928 06:31:18.194380 6106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
I20260928 06:31:18.195592 6238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:18.195106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36479 (local address 127.22.225.131:60857)
0928 06:31:18.195279 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:18.195391 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:36479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":71,"threads_started":1}
W20260928 06:31:18.196049 6236 heartbeater.cc:644] Failed to heartbeat to 127.22.225.190:36479 (0 consecutive failures): Network error: Failed to ping master at 127.22.225.190:36479: Client connection negotiation failed: client connection to 127.22.225.190:36479: connect: Connection refused (error 111)
I20260928 06:31:18.196362 6238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:18.196257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36479 (local address 127.22.225.131:54455)
0928 06:31:18.196305 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:18.196339 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:36479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 06:31:18.196715 6238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:18.196631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36479 (local address 127.22.225.131:59789)
0928 06:31:18.196668 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:18.196697 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:36479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260928 06:31:18.196834 6236 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 06:31:18.197053 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 6106
I20260928 06:31:18.975525 6240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:18.975267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36479 (local address 127.22.225.129:56455)
0928 06:31:18.975406 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:18.975471 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:36479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":69,"threads_started":1}
I20260928 06:31:19.057683 6241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:19.057459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36479 (local address 127.22.225.130:56429)
0928 06:31:19.057586 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:19.057623 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:36479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":76,"threads_started":1}
I20260928 06:31:19.197422 6242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:19.197208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36479 (local address 127.22.225.131:34461)
0928 06:31:19.197333 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:19.197368 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:36479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":74,"threads_started":1}
I20260928 06:31:19.211087 6243 raft_consensus.cc:482] T d4b959b495d944bcbc51d50c7f194871 P 25bfab18c92d4f6cb8727ef443a59867 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:31:19.211236 6243 raft_consensus.cc:504] T d4b959b495d944bcbc51d50c7f194871 P 25bfab18c92d4f6cb8727ef443a59867 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f7f1fcfb1f4202973d15f5827e8d13" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 38549 } } peers { permanent_uuid: "cf5ca4308f864408828089cc6a5532af" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41535 } } peers { permanent_uuid: "25bfab18c92d4f6cb8727ef443a59867" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34211 } }
I20260928 06:31:19.211611 6243 leader_election.cc:288] T d4b959b495d944bcbc51d50c7f194871 P 25bfab18c92d4f6cb8727ef443a59867 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 87f7f1fcfb1f4202973d15f5827e8d13 (127.22.225.131:38549), cf5ca4308f864408828089cc6a5532af (127.22.225.130:41535)
Sep 28 06:31:19 dist-test-slave-d59m krb5kdc[4666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577077, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.129@KRBTEST.COM for kudu/127.22.225.130@KRBTEST.COM
Sep 28 06:31:19 dist-test-slave-d59m krb5kdc[4666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577077, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.129@KRBTEST.COM for kudu/127.22.225.131@KRBTEST.COM
I20260928 06:31:19.223423 6240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:19.211747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.131:38549 (local address 127.22.225.129:35553)
0928 06:31:19.211802 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:19.211816 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:19.212068 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:19.212895 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:19.212896 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:19.213137 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:19.213315 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:19.213322 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:19.214762 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:19.214763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:19.215518 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:19.215520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:19.216334 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:19.216335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:19.216384 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:19.217116 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:19.217127 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:19.219449 (+ 2322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:19.221766 (+ 2317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:19.221767 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:19.221768 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:19.222043 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:19.222296 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:19.222297 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:19.222298 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:19.222387 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:19.222840 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:19.222846 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:19.223008 (+ 162us) client_negotiation.cc:769] Sending connection context
0928 06:31:19.223123 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 06:31:19.223326 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":2}
I20260928 06:31:19.223436 6245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:19.211919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:35553 (local address 127.22.225.131:38549)
0928 06:31:19.212056 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:19.212071 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:19.212117 (+ 46us) server_negotiation.cc:407] Connection header received
0928 06:31:19.212713 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:19.212717 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:19.212766 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:19.212904 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:19.213441 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:19.214670 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:19.215608 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:19.216256 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:19.216395 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:19.219555 (+ 3160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:19.219559 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:19.219560 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:19.219572 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:19.221677 (+ 2105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:19.222131 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:19.222133 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:19.222134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:19.222240 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:19.222480 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:19.222481 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:19.222481 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:19.222762 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:19.222851 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:19.223177 (+ 326us) server_negotiation.cc:299] Negotiation successful
0928 06:31:19.223282 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":63,"threads_started":1}
I20260928 06:31:19.224376 6244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:19.211884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.130:41535 (local address 127.22.225.129:43843)
0928 06:31:19.212156 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:19.212167 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:19.212222 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:19.212642 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:19.212645 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:19.213124 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:19.213314 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:19.213320 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:19.214421 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:19.214422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:19.215514 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:19.215516 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:19.216218 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:19.216219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:19.216355 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:19.217144 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:19.217148 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:19.219057 (+ 1909us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:19.221569 (+ 2512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:19.221577 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:19.221580 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:19.221906 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:19.222228 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:19.222229 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:19.222229 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:19.222362 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:19.224075 (+ 1713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:19.224076 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:19.224121 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 06:31:19.224200 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 06:31:19.224290 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":3,"thread_start_us":69,"threads_started":1}
I20260928 06:31:19.224535 6246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:19.212011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:43843 (local address 127.22.225.130:41535)
0928 06:31:19.212200 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:19.212211 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:19.212236 (+ 25us) server_negotiation.cc:407] Connection header received
0928 06:31:19.212377 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:19.212380 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:19.212429 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:19.212554 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:19.213429 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:19.214311 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:19.215639 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:19.216076 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:19.216202 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:19.219183 (+ 2981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:19.219189 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:19.219189 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:19.219202 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:19.221464 (+ 2262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:19.221990 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:19.221993 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:19.221994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:19.222154 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:19.222434 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:19.222435 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:19.222436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:19.222762 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:19.222851 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:19.224281 (+ 1430us) server_negotiation.cc:299] Negotiation successful
0928 06:31:19.224426 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":55,"threads_started":1}
W20260928 06:31:19.224637 5856 leader_election.cc:341] T d4b959b495d944bcbc51d50c7f194871 P 25bfab18c92d4f6cb8727ef443a59867 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 87f7f1fcfb1f4202973d15f5827e8d13 (127.22.225.131:38549): Not found: Tablet not found: d4b959b495d944bcbc51d50c7f194871
I20260928 06:31:19.225234 6056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4b959b495d944bcbc51d50c7f194871" candidate_uuid: "25bfab18c92d4f6cb8727ef443a59867" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "cf5ca4308f864408828089cc6a5532af" is_pre_election: true
I20260928 06:31:19.225392 6056 raft_consensus.cc:2454] T d4b959b495d944bcbc51d50c7f194871 P cf5ca4308f864408828089cc6a5532af [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25bfab18c92d4f6cb8727ef443a59867 in term 1.
I20260928 06:31:19.225639 5856 leader_election.cc:302] T d4b959b495d944bcbc51d50c7f194871 P 25bfab18c92d4f6cb8727ef443a59867 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 25bfab18c92d4f6cb8727ef443a59867, cf5ca4308f864408828089cc6a5532af; no voters: 87f7f1fcfb1f4202973d15f5827e8d13
I20260928 06:31:19.225757 6243 raft_consensus.cc:2781] T d4b959b495d944bcbc51d50c7f194871 P 25bfab18c92d4f6cb8727ef443a59867 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 06:31:19.225819 6243 raft_consensus.cc:482] T d4b959b495d944bcbc51d50c7f194871 P 25bfab18c92d4f6cb8727ef443a59867 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:31:19.225844 6243 raft_consensus.cc:3037] T d4b959b495d944bcbc51d50c7f194871 P 25bfab18c92d4f6cb8727ef443a59867 [term 1 FOLLOWER]: Advancing to term 2
I20260928 06:31:19.226994 6243 raft_consensus.cc:504] T d4b959b495d944bcbc51d50c7f194871 P 25bfab18c92d4f6cb8727ef443a59867 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f7f1fcfb1f4202973d15f5827e8d13" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 38549 } } peers { permanent_uuid: "cf5ca4308f864408828089cc6a5532af" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41535 } } peers { permanent_uuid: "25bfab18c92d4f6cb8727ef443a59867" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34211 } }
I20260928 06:31:19.227141 6243 leader_election.cc:288] T d4b959b495d944bcbc51d50c7f194871 P 25bfab18c92d4f6cb8727ef443a59867 [CANDIDATE]: Term 2 election: Requested vote from peers 87f7f1fcfb1f4202973d15f5827e8d13 (127.22.225.131:38549), cf5ca4308f864408828089cc6a5532af (127.22.225.130:41535)
I20260928 06:31:19.227329 6056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4b959b495d944bcbc51d50c7f194871" candidate_uuid: "25bfab18c92d4f6cb8727ef443a59867" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "cf5ca4308f864408828089cc6a5532af"
I20260928 06:31:19.227419 6056 raft_consensus.cc:3037] T d4b959b495d944bcbc51d50c7f194871 P cf5ca4308f864408828089cc6a5532af [term 1 FOLLOWER]: Advancing to term 2
W20260928 06:31:19.227468 5856 leader_election.cc:341] T d4b959b495d944bcbc51d50c7f194871 P 25bfab18c92d4f6cb8727ef443a59867 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 87f7f1fcfb1f4202973d15f5827e8d13 (127.22.225.131:38549): Not found: Tablet not found: d4b959b495d944bcbc51d50c7f194871
I20260928 06:31:19.228490 6056 raft_consensus.cc:2454] T d4b959b495d944bcbc51d50c7f194871 P cf5ca4308f864408828089cc6a5532af [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25bfab18c92d4f6cb8727ef443a59867 in term 2.
I20260928 06:31:19.228714 5856 leader_election.cc:302] T d4b959b495d944bcbc51d50c7f194871 P 25bfab18c92d4f6cb8727ef443a59867 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 25bfab18c92d4f6cb8727ef443a59867, cf5ca4308f864408828089cc6a5532af; no voters: 87f7f1fcfb1f4202973d15f5827e8d13
I20260928 06:31:19.228853 6243 raft_consensus.cc:2781] T d4b959b495d944bcbc51d50c7f194871 P 25bfab18c92d4f6cb8727ef443a59867 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 06:31:19.228984 6243 raft_consensus.cc:686] T d4b959b495d944bcbc51d50c7f194871 P 25bfab18c92d4f6cb8727ef443a59867 [term 2 LEADER]: Becoming Leader. State: Replica: 25bfab18c92d4f6cb8727ef443a59867, State: Running, Role: LEADER
I20260928 06:31:19.229081 6243 consensus_queue.cc:237] T d4b959b495d944bcbc51d50c7f194871 P 25bfab18c92d4f6cb8727ef443a59867 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f7f1fcfb1f4202973d15f5827e8d13" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 38549 } } peers { permanent_uuid: "cf5ca4308f864408828089cc6a5532af" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41535 } } peers { permanent_uuid: "25bfab18c92d4f6cb8727ef443a59867" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34211 } }
I20260928 06:31:19.229290 6244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:19.229188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36479 (local address 127.22.225.129:37265)
0928 06:31:19.229234 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:19.229263 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:36479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 06:31:19.739704 6056 raft_consensus.cc:1261] T d4b959b495d944bcbc51d50c7f194871 P cf5ca4308f864408828089cc6a5532af [term 2 FOLLOWER]: Refusing update from remote peer 25bfab18c92d4f6cb8727ef443a59867: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260928 06:31:19.740155 6249 consensus_queue.cc:1035] T d4b959b495d944bcbc51d50c7f194871 P 25bfab18c92d4f6cb8727ef443a59867 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf5ca4308f864408828089cc6a5532af" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260928 06:31:19.741178 6252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:19.740973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36479 (local address 127.22.225.130:33769)
0928 06:31:19.741082 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:19.741136 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:36479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":58,"threads_started":1}
W20260928 06:31:19.742228 5856 consensus_peers.cc:487] T d4b959b495d944bcbc51d50c7f194871 P 25bfab18c92d4f6cb8727ef443a59867 -> Peer 87f7f1fcfb1f4202973d15f5827e8d13 (127.22.225.131:38549): Couldn't send request to peer 87f7f1fcfb1f4202973d15f5827e8d13. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d4b959b495d944bcbc51d50c7f194871. This is attempt 1: this message will repeat every 5th retry.
I20260928 06:31:19.742306 6257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:19.742069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36479 (local address 127.22.225.129:52749)
0928 06:31:19.742230 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:19.742271 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:36479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":105,"threads_started":1}
I20260928 06:31:19.742297 6252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:19.742204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36479 (local address 127.22.225.130:58285)
0928 06:31:19.742242 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:19.742265 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:36479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 06:31:20.170814 6258 ts_tablet_manager.cc:927] T d4b959b495d944bcbc51d50c7f194871 P 87f7f1fcfb1f4202973d15f5827e8d13: Initiating tablet copy from peer 25bfab18c92d4f6cb8727ef443a59867 (127.22.225.129:34211)
I20260928 06:31:20.171036 6258 tablet_copy_client.cc:316] T d4b959b495d944bcbc51d50c7f194871 P 87f7f1fcfb1f4202973d15f5827e8d13: tablet copy: Beginning tablet copy session from remote peer at address 127.22.225.129:34211
Sep 28 06:31:20 dist-test-slave-d59m krb5kdc[4666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577078, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.131@KRBTEST.COM for kudu/127.22.225.129@KRBTEST.COM
I20260928 06:31:20.179257 6261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:20.171453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:57933 (local address 127.22.225.129:34211)
0928 06:31:20.171655 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:20.171662 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:20.171710 (+ 48us) server_negotiation.cc:407] Connection header received
0928 06:31:20.171738 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:20.171741 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:20.171758 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:20.171837 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:20.172332 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:20.173104 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:20.173907 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:20.174212 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:20.174299 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:20.176400 (+ 2101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:20.176407 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:20.176408 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:20.176415 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:20.177982 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:20.178371 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:20.178375 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:20.178375 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:20.178437 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:20.178631 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:20.178632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:20.178632 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:20.178782 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:20.178869 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:20.179064 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 06:31:20.179151 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":108,"threads_started":1}
I20260928 06:31:20.179267 6260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:20.171360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.129:34211 (local address 127.22.225.131:57933)
0928 06:31:20.171550 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:20.171570 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:20.171649 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:20.171848 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:20.171855 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:20.172179 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:20.172246 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:20.172248 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:20.173208 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:20.173209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:20.173822 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:20.173824 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:20.174299 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:20.174301 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:20.174392 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:20.174925 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:20.174929 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:20.176296 (+ 1367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:20.178067 (+ 1771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:20.178074 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:20.178077 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:20.178288 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:20.178504 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:20.178505 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:20.178506 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:20.178570 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:20.178859 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:20.178865 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:20.178919 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 06:31:20.179027 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 06:31:20.179151 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":114,"threads_started":1}
I20260928 06:31:20.179711 5941 tablet_copy_service.cc:140] P 25bfab18c92d4f6cb8727ef443a59867: Received BeginTabletCopySession request for tablet d4b959b495d944bcbc51d50c7f194871 from peer 87f7f1fcfb1f4202973d15f5827e8d13 ({username='kudu', principal='kudu/127.22.225.131@KRBTEST.COM'} at 127.22.225.131:57933)
I20260928 06:31:20.179795 5941 tablet_copy_service.cc:161] P 25bfab18c92d4f6cb8727ef443a59867: Beginning new tablet copy session on tablet d4b959b495d944bcbc51d50c7f194871 from peer 87f7f1fcfb1f4202973d15f5827e8d13 at {username='kudu', principal='kudu/127.22.225.131@KRBTEST.COM'} at 127.22.225.131:57933: session id = 87f7f1fcfb1f4202973d15f5827e8d13-d4b959b495d944bcbc51d50c7f194871
I20260928 06:31:20.180369 5941 tablet_copy_source_session.cc:205] T d4b959b495d944bcbc51d50c7f194871 P 25bfab18c92d4f6cb8727ef443a59867: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 06:31:20.181371 6258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4b959b495d944bcbc51d50c7f194871. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:20.184134 6258 tablet_copy_client.cc:791] T d4b959b495d944bcbc51d50c7f194871 P 87f7f1fcfb1f4202973d15f5827e8d13: tablet copy: Starting download of 0 data blocks...
I20260928 06:31:20.184291 6258 tablet_copy_client.cc:655] T d4b959b495d944bcbc51d50c7f194871 P 87f7f1fcfb1f4202973d15f5827e8d13: tablet copy: Starting download of 1 WAL segments...
I20260928 06:31:20.185185 6258 tablet_copy_client.cc:531] T d4b959b495d944bcbc51d50c7f194871 P 87f7f1fcfb1f4202973d15f5827e8d13: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 06:31:20.185858 6258 tablet_bootstrap.cc:492] T d4b959b495d944bcbc51d50c7f194871 P 87f7f1fcfb1f4202973d15f5827e8d13: Bootstrap starting.
I20260928 06:31:20.187283 6258 log.cc:824] T d4b959b495d944bcbc51d50c7f194871 P 87f7f1fcfb1f4202973d15f5827e8d13: Log is configured to *not* fsync() on all Append() calls
I20260928 06:31:20.189778 6258 tablet_bootstrap.cc:492] T d4b959b495d944bcbc51d50c7f194871 P 87f7f1fcfb1f4202973d15f5827e8d13: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:31:20.190007 6258 tablet_bootstrap.cc:492] T d4b959b495d944bcbc51d50c7f194871 P 87f7f1fcfb1f4202973d15f5827e8d13: Bootstrap complete.
I20260928 06:31:20.190160 6258 ts_tablet_manager.cc:1397] T d4b959b495d944bcbc51d50c7f194871 P 87f7f1fcfb1f4202973d15f5827e8d13: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260928 06:31:20.191200 6258 raft_consensus.cc:348] T d4b959b495d944bcbc51d50c7f194871 P 87f7f1fcfb1f4202973d15f5827e8d13 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f7f1fcfb1f4202973d15f5827e8d13" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 38549 } } peers { permanent_uuid: "cf5ca4308f864408828089cc6a5532af" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41535 } } peers { permanent_uuid: "25bfab18c92d4f6cb8727ef443a59867" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34211 } }
I20260928 06:31:20.191358 6258 raft_consensus.cc:729] T d4b959b495d944bcbc51d50c7f194871 P 87f7f1fcfb1f4202973d15f5827e8d13 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87f7f1fcfb1f4202973d15f5827e8d13, State: Initialized, Role: FOLLOWER
I20260928 06:31:20.191514 6258 consensus_queue.cc:260] T d4b959b495d944bcbc51d50c7f194871 P 87f7f1fcfb1f4202973d15f5827e8d13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f7f1fcfb1f4202973d15f5827e8d13" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 38549 } } peers { permanent_uuid: "cf5ca4308f864408828089cc6a5532af" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 41535 } } peers { permanent_uuid: "25bfab18c92d4f6cb8727ef443a59867" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34211 } }
I20260928 06:31:20.191784 6258 ts_tablet_manager.cc:1428] T d4b959b495d944bcbc51d50c7f194871 P 87f7f1fcfb1f4202973d15f5827e8d13: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 06:31:20.192073 6260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:20.191976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36479 (local address 127.22.225.131:59345)
0928 06:31:20.192011 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:20.192041 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:36479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260928 06:31:20.192059 5941 tablet_copy_service.cc:342] P 25bfab18c92d4f6cb8727ef443a59867: Request end of tablet copy session 87f7f1fcfb1f4202973d15f5827e8d13-d4b959b495d944bcbc51d50c7f194871 received from {username='kudu', principal='kudu/127.22.225.131@KRBTEST.COM'} at 127.22.225.131:57933
I20260928 06:31:20.192135 5941 tablet_copy_service.cc:433] P 25bfab18c92d4f6cb8727ef443a59867: ending tablet copy session 87f7f1fcfb1f4202973d15f5827e8d13-d4b959b495d944bcbc51d50c7f194871 on tablet d4b959b495d944bcbc51d50c7f194871 with peer 87f7f1fcfb1f4202973d15f5827e8d13
I20260928 06:31:20.192538 6260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:20.192443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36479 (local address 127.22.225.131:39897)
0928 06:31:20.192491 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:20.192515 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:36479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 06:31:20.742938 6265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:20.742737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36479 (local address 127.22.225.129:54293)
0928 06:31:20.742851 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:20.742886 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:36479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":49,"threads_started":1}
I20260928 06:31:20.742938 6264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:20.742721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36479 (local address 127.22.225.130:40903)
0928 06:31:20.742857 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:20.742885 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:36479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":66,"threads_started":1}
I20260928 06:31:20.788851 6265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:20.788692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36479 (local address 127.22.225.129:39223)
0928 06:31:20.788757 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:20.788801 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:36479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 06:31:20.788810 6190 raft_consensus.cc:1203] T d4b959b495d944bcbc51d50c7f194871 P 87f7f1fcfb1f4202973d15f5827e8d13 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260928 06:31:20.789525 6268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:20.789255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36479 (local address 127.22.225.131:59275)
0928 06:31:20.789430 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:20.789465 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:36479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":55,"threads_started":1}
I20260928 06:31:21.743780 6269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:21.743366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36479 (local address 127.22.225.130:60499)
0928 06:31:21.743644 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:21.743714 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:36479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":67,"threads_started":1}
I20260928 06:31:21.789700 6270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:21.789327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36479 (local address 127.22.225.129:44701)
0928 06:31:21.789617 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:21.789653 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:36479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":239,"threads_started":1}
I20260928 06:31:21.790146 6271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:21.789936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36479 (local address 127.22.225.131:45295)
0928 06:31:21.790055 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:21.790099 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:36479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":50,"threads_started":1}
I20260928 06:31:22.744690 6272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:22.744385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36479 (local address 127.22.225.130:37897)
0928 06:31:22.744559 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:22.744638 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:36479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":80,"threads_started":1}
I20260928 06:31:22.790467 6273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:22.790226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36479 (local address 127.22.225.129:51535)
0928 06:31:22.790352 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:22.790389 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:36479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":75,"threads_started":1}
I20260928 06:31:22.790745 6274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:22.790548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36479 (local address 127.22.225.131:33121)
0928 06:31:22.790686 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:22.790710 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:36479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":48,"threads_started":1}
I20260928 06:31:23.745395 6275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:23.745177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36479 (local address 127.22.225.130:60173)
0928 06:31:23.745310 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:23.745350 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:36479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":77,"threads_started":1}
I20260928 06:31:23.791467 6276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:23.791225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36479 (local address 127.22.225.131:41831)
0928 06:31:23.791373 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:23.791413 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:36479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":69,"threads_started":1}
I20260928 06:31:23.791467 6277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:23.791225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36479 (local address 127.22.225.129:48191)
0928 06:31:23.791374 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:23.791413 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:36479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":71,"threads_started":1}
I20260928 06:31:23.791810 6277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:23.791727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36479 (local address 127.22.225.129:59623)
0928 06:31:23.791760 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:23.791788 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:36479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260928 06:31:24.746104 6278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:24.745859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36479 (local address 127.22.225.130:56659)
0928 06:31:24.745998 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:24.746041 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:36479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":66,"threads_started":1}
I20260928 06:31:24.792296 6279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:24.792006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36479 (local address 127.22.225.131:59981)
0928 06:31:24.792163 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:24.792225 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:36479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":89,"threads_started":1}
I20260928 06:31:24.792608 6280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:24.792241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36479 (local address 127.22.225.129:33677)
0928 06:31:24.792485 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:24.792532 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:36479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":79,"threads_started":1}
I20260928 06:31:25.746842 6281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:25.746572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36479 (local address 127.22.225.130:39689)
0928 06:31:25.746718 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:25.746780 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:36479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":78,"threads_started":1}
I20260928 06:31:25.793058 6282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:25.792826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36479 (local address 127.22.225.131:42037)
0928 06:31:25.792963 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:25.793012 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:36479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":81,"threads_started":1}
I20260928 06:31:25.793227 6283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:25.793019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36479 (local address 127.22.225.129:58083)
0928 06:31:25.793148 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:25.793177 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:36479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":68,"threads_started":1}
I20260928 06:31:26.747668 6284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:26.747427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36479 (local address 127.22.225.130:50269)
0928 06:31:26.747570 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:26.747608 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:36479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":86,"threads_started":1}
I20260928 06:31:26.793790 6285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:26.793527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36479 (local address 127.22.225.131:42393)
0928 06:31:26.793688 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:26.793734 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:36479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":87,"threads_started":1}
I20260928 06:31:26.793931 6286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:26.793670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36479 (local address 127.22.225.129:43009)
0928 06:31:26.793784 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:26.793824 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:36479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":67,"threads_started":1}
I20260928 06:31:27.748507 6287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:27.748222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36479 (local address 127.22.225.130:57963)
0928 06:31:27.748370 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:27.748412 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:36479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":83,"threads_started":1}
I20260928 06:31:27.794471 6288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:27.794222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36479 (local address 127.22.225.131:33051)
0928 06:31:27.794360 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:27.794405 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:36479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":75,"threads_started":1}
I20260928 06:31:27.794718 6289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:27.794482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36479 (local address 127.22.225.129:55851)
0928 06:31:27.794634 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:27.794667 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:36479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":56,"threads_started":1}
I20260928 06:31:28.749181 6290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:28.748973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36479 (local address 127.22.225.130:56849)
0928 06:31:28.749101 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:28.749135 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:36479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":71,"threads_started":1}
I20260928 06:31:28.795337 6292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:28.795135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36479 (local address 127.22.225.129:40367)
0928 06:31:28.795249 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:28.795291 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:36479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":62,"threads_started":1}
I20260928 06:31:28.795367 6291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:28.795100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36479 (local address 127.22.225.131:37545)
0928 06:31:28.795249 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:28.795303 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:36479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":70,"threads_started":1}
I20260928 06:31:29.749853 6293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:29.749654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36479 (local address 127.22.225.130:53745)
0928 06:31:29.749777 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:29.749807 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:36479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":60,"threads_started":1}
I20260928 06:31:29.796149 6295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:29.795912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36479 (local address 127.22.225.129:56087)
0928 06:31:29.796068 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:29.796103 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:36479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":68,"threads_started":1}
I20260928 06:31:29.796156 6294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:29.795881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36479 (local address 127.22.225.131:56099)
0928 06:31:29.796054 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:29.796106 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:36479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":96,"threads_started":1}
I20260928 06:31:30.750535 6296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:30.750296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36479 (local address 127.22.225.130:46801)
0928 06:31:30.750449 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:30.750487 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:36479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":93,"threads_started":1}
I20260928 06:31:30.796991 6297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:30.796741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36479 (local address 127.22.225.131:46151)
0928 06:31:30.796894 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:30.796939 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:36479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":71,"threads_started":1}
I20260928 06:31:30.797003 6298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:30.796743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36479 (local address 127.22.225.129:39347)
0928 06:31:30.796910 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:30.796950 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:36479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":96,"threads_started":1}
I20260928 06:31:31.751385 6299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:31.751137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36479 (local address 127.22.225.130:41917)
0928 06:31:31.751292 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:31.751337 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:36479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":82,"threads_started":1}
I20260928 06:31:31.797765 6300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:31.797512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36479 (local address 127.22.225.129:55963)
0928 06:31:31.797670 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:31.797713 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:36479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":72,"threads_started":1}
I20260928 06:31:31.797765 6301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:31.797478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36479 (local address 127.22.225.131:47897)
0928 06:31:31.797672 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:31.797710 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:36479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":103,"threads_started":1}
I20260928 06:31:32.752184 6302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:32.751914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36479 (local address 127.22.225.130:45791)
0928 06:31:32.752063 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:32.752117 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:36479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":54,"threads_started":1}
I20260928 06:31:32.798403 6303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:32.798189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36479 (local address 127.22.225.131:58737)
0928 06:31:32.798316 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:32.798355 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:36479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":84,"threads_started":1}
I20260928 06:31:32.798448 6304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:32.798238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36479 (local address 127.22.225.129:56303)
0928 06:31:32.798352 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:32.798390 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:36479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":67,"threads_started":1}
I20260928 06:31:33.198817 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 5808
I20260928 06:31:33.204671 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 5972
I20260928 06:31:33.210062 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 6106
2026-09-28T06:31:33Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18453 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:31:33 dist-test-slave-d59m krb5kdc[6307](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:31:33 dist-test-slave-d59m krb5kdc[6307](info): set up 2 sockets
Sep 28 06:31:33 dist-test-slave-d59m krb5kdc[6307](info): commencing operation
krb5kdc: starting...
W20260928 06:31:34.301403 23430 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Sep 28 06:31:34 dist-test-slave-d59m krb5kdc[6307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:31:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:31:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab.
Entry for principal kudu/127.22.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab.
I20260928 06:31:34.426968 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:39765
--webserver_interface=127.22.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab
--principal=kudu/127.22.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:38327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.225.190:39765
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:31:34.533672 6320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:34.533943 6320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:34.533988 6320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:34.535271 6320 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 06:31:34.535347 6320 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 06:31:34.535377 6320 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 06:31:34.535411 6320 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:31:34.535442 6320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:34.535475 6320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:34.535511 6320 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:31:34.535544 6320 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:31:34.537282 6320 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:38327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.22.225.190:39765
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab
--principal=kudu/127.22.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:39765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:31:34.537606 6320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:34.537864 6320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:34.543268 6328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:34.543365 6325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:34.543462 6326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:34.543398 6320 server_base.cc:1034] running on GCE node
I20260928 06:31:34.544083 6320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:34.544404 6320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:34.545573 6320 hybrid_clock.cc:648] HybridClock initialized: now 1790577094545551 us; error 38 us; skew 500 ppm
Sep 28 06:31:34 dist-test-slave-d59m krb5kdc[6307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577094, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:34.548458 6320 init.cc:377] Logged in from keytab as kudu/127.22.225.190@KRBTEST.COM (short username kudu)
I20260928 06:31:34.548928 6320 webserver.cc:460] Webserver started at http://127.22.225.190:45977/ using document root <none> and password file <none>
I20260928 06:31:34.549187 6320 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:34.549258 6320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:34.549407 6320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:34.550480 6320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/master-0/data/instance:
uuid: "c3eacb4d322f415eb7bb4b021abfbece"
format_stamp: "Formatted at 2026-09-28 06:31:34 on dist-test-slave-d59m"
server_key: "0ae844b205141afacca93022773b09dd"
server_key_iv: "88b2d78127515b9607f76597944b916c"
server_key_version: "encryptionkey@0"
I20260928 06:31:34.550849 6320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/master-0/wal/instance:
uuid: "c3eacb4d322f415eb7bb4b021abfbece"
format_stamp: "Formatted at 2026-09-28 06:31:34 on dist-test-slave-d59m"
server_key: "0ae844b205141afacca93022773b09dd"
server_key_iv: "88b2d78127515b9607f76597944b916c"
server_key_version: "encryptionkey@0"
I20260928 06:31:34.552244 6320 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:31:34.553148 6335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:34.553447 6320 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:31:34.553548 6320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/master-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/master-0/wal
uuid: "c3eacb4d322f415eb7bb4b021abfbece"
format_stamp: "Formatted at 2026-09-28 06:31:34 on dist-test-slave-d59m"
server_key: "0ae844b205141afacca93022773b09dd"
server_key_iv: "88b2d78127515b9607f76597944b916c"
server_key_version: "encryptionkey@0"
I20260928 06:31:34.553637 6320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:34.563490 6320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:34.565913 6320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:34.566087 6320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:34.571020 6320 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.190:39765
I20260928 06:31:34.571062 6387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.190:39765 every 8 connection(s)
I20260928 06:31:34.571419 6320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
I20260928 06:31:34.572314 6388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:34.572786 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 6320
I20260928 06:31:34.572888 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/master-0/wal/instance
I20260928 06:31:34.573110 23430 external_mini_cluster.cc:1428] Setting key 20c26e982f3e30d0e6831a085d1123f7
I20260928 06:31:34.575325 6388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3eacb4d322f415eb7bb4b021abfbece: Bootstrap starting.
I20260928 06:31:34.575938 6388 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3eacb4d322f415eb7bb4b021abfbece: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:34.576236 6388 log.cc:824] T 00000000000000000000000000000000 P c3eacb4d322f415eb7bb4b021abfbece: Log is configured to *not* fsync() on all Append() calls
I20260928 06:31:34.577080 6388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3eacb4d322f415eb7bb4b021abfbece: No bootstrap required, opened a new log
I20260928 06:31:34.579413 6388 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3eacb4d322f415eb7bb4b021abfbece [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3eacb4d322f415eb7bb4b021abfbece" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 39765 } }
I20260928 06:31:34.579608 6388 raft_consensus.cc:374] T 00000000000000000000000000000000 P c3eacb4d322f415eb7bb4b021abfbece [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:34.579649 6388 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3eacb4d322f415eb7bb4b021abfbece [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3eacb4d322f415eb7bb4b021abfbece, State: Initialized, Role: FOLLOWER
I20260928 06:31:34.579845 6388 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3eacb4d322f415eb7bb4b021abfbece [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3eacb4d322f415eb7bb4b021abfbece" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 39765 } }
I20260928 06:31:34.579946 6388 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3eacb4d322f415eb7bb4b021abfbece [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:31:34.580016 6388 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3eacb4d322f415eb7bb4b021abfbece [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:31:34.580078 6388 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3eacb4d322f415eb7bb4b021abfbece [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:34.580824 6388 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3eacb4d322f415eb7bb4b021abfbece [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3eacb4d322f415eb7bb4b021abfbece" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 39765 } }
I20260928 06:31:34.580945 6388 leader_election.cc:302] T 00000000000000000000000000000000 P c3eacb4d322f415eb7bb4b021abfbece [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3eacb4d322f415eb7bb4b021abfbece; no voters:
Sep 28 06:31:34 dist-test-slave-d59m krb5kdc[6307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:31:34.581113 6388 leader_election.cc:288] T 00000000000000000000000000000000 P c3eacb4d322f415eb7bb4b021abfbece [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:31:34.581163 6393 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3eacb4d322f415eb7bb4b021abfbece [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:31:34.581338 6393 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3eacb4d322f415eb7bb4b021abfbece [term 1 LEADER]: Becoming Leader. State: Replica: c3eacb4d322f415eb7bb4b021abfbece, State: Running, Role: LEADER
I20260928 06:31:34.581436 6388 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3eacb4d322f415eb7bb4b021abfbece [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:31:34.581434 6393 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3eacb4d322f415eb7bb4b021abfbece [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3eacb4d322f415eb7bb4b021abfbece" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 39765 } }
I20260928 06:31:34.581809 6393 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3eacb4d322f415eb7bb4b021abfbece [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3eacb4d322f415eb7bb4b021abfbece. Latest consensus state: current_term: 1 leader_uuid: "c3eacb4d322f415eb7bb4b021abfbece" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3eacb4d322f415eb7bb4b021abfbece" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 39765 } } }
I20260928 06:31:34.581892 6393 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3eacb4d322f415eb7bb4b021abfbece [sys.catalog]: This master's current role is: LEADER
I20260928 06:31:34.581869 6395 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3eacb4d322f415eb7bb4b021abfbece [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3eacb4d322f415eb7bb4b021abfbece" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3eacb4d322f415eb7bb4b021abfbece" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 39765 } } }
I20260928 06:31:34.581970 6395 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3eacb4d322f415eb7bb4b021abfbece [sys.catalog]: This master's current role is: LEADER
I20260928 06:31:34.582125 6397 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:31:34.582696 6397 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:31:34.584260 6397 catalog_manager.cc:1347] Generated new cluster ID: ec1aca84d3834c95a23cf3dea47cef2e
I20260928 06:31:34.584314 6397 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:31:34.586861 6391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:34.573708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47200 (local address 127.22.225.190:39765)
0928 06:31:34.573904 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:34.573914 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:34.573951 (+ 37us) server_negotiation.cc:407] Connection header received
0928 06:31:34.574489 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:34.574497 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:34.574557 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:34.574716 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:34.575459 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:34.576789 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:34.577866 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:34.578465 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:34.578585 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:34.581595 (+ 3010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:34.581599 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:34.581600 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:34.581611 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:34.584923 (+ 3312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:34.585337 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:34.585340 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:34.585340 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:34.585442 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:34.585634 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:34.585635 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:34.585635 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:34.585924 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:34.586078 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:34.586531 (+ 453us) server_negotiation.cc:299] Negotiation successful
0928 06:31:34.586663 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":80,"threads_started":1}
I20260928 06:31:34.592137 6397 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:31:34.592916 6397 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:31:34.602586 6397 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c3eacb4d322f415eb7bb4b021abfbece: Generated new TSK 0
I20260928 06:31:34.602818 6397 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.225.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab.
Entry for principal kudu/127.22.225.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab.
I20260928 06:31:34.631735 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.129:0
--local_ip_for_outbound_sockets=127.22.225.129
--webserver_interface=127.22.225.129
--webserver_port=0
--tserver_master_addrs=127.22.225.190:39765
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab
--principal=kudu/127.22.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:38327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:31:34.751241 6414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:34.751533 6414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:34.751575 6414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:34.753314 6414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:34.753381 6414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:34.753423 6414 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.129
I20260928 06:31:34.755565 6414 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:38327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab
--principal=kudu/127.22.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.225.129
--webserver_port=0
--tserver_master_addrs=127.22.225.190:39765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.129
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:31:34.755818 6414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:34.756119 6414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:34.761556 6420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:34.761667 6419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:34.761567 6422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:34.761808 6414 server_base.cc:1034] running on GCE node
I20260928 06:31:34.762056 6414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:34.762290 6414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:34.763536 6414 hybrid_clock.cc:648] HybridClock initialized: now 1790577094763515 us; error 33 us; skew 500 ppm
Sep 28 06:31:34 dist-test-slave-d59m krb5kdc[6307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577094, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:34.766877 6414 init.cc:377] Logged in from keytab as kudu/127.22.225.129@KRBTEST.COM (short username kudu)
I20260928 06:31:34.767392 6414 webserver.cc:460] Webserver started at http://127.22.225.129:43921/ using document root <none> and password file <none>
I20260928 06:31:34.767685 6414 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:34.767786 6414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:34.767918 6414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:34.769315 6414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-0/data/instance:
uuid: "43a313fadad1418cbcebf59dd0ecb76a"
format_stamp: "Formatted at 2026-09-28 06:31:34 on dist-test-slave-d59m"
server_key: "ed44e41bb9338f5afaa97a8b586735cd"
server_key_iv: "ae625cd819ee12cd936dcbf37b7c62df"
server_key_version: "encryptionkey@0"
I20260928 06:31:34.769744 6414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-0/wal/instance:
uuid: "43a313fadad1418cbcebf59dd0ecb76a"
format_stamp: "Formatted at 2026-09-28 06:31:34 on dist-test-slave-d59m"
server_key: "ed44e41bb9338f5afaa97a8b586735cd"
server_key_iv: "ae625cd819ee12cd936dcbf37b7c62df"
server_key_version: "encryptionkey@0"
I20260928 06:31:34.771381 6414 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:31:34.772246 6429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:34.772409 6414 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:34.772482 6414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-0/wal
uuid: "43a313fadad1418cbcebf59dd0ecb76a"
format_stamp: "Formatted at 2026-09-28 06:31:34 on dist-test-slave-d59m"
server_key: "ed44e41bb9338f5afaa97a8b586735cd"
server_key_iv: "ae625cd819ee12cd936dcbf37b7c62df"
server_key_version: "encryptionkey@0"
I20260928 06:31:34.772552 6414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:34.787645 6414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:34.789819 6414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:34.789983 6414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:34.790235 6414 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:31:34.790577 6414 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:31:34.790635 6414 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:34.790697 6414 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:31:34.790726 6414 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:34.796365 6414 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.129:34637
I20260928 06:31:34.796464 6542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.129:34637 every 8 connection(s)
I20260928 06:31:34.796800 6414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
I20260928 06:31:34.798233 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 6414
I20260928 06:31:34.798346 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-0/wal/instance
I20260928 06:31:34.798600 23430 external_mini_cluster.cc:1428] Setting key c76ece319319a570d08350a1724d1fe7
Sep 28 06:31:34 dist-test-slave-d59m krb5kdc[6307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577094, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.129@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:31:34.807381 6391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:34.797560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:40353 (local address 127.22.225.190:39765)
0928 06:31:34.797629 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:34.797631 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:34.797870 (+ 239us) server_negotiation.cc:407] Connection header received
0928 06:31:34.798529 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:34.798539 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:34.798554 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:34.798614 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:34.800001 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:34.800771 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:34.801795 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:34.802021 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:34.802114 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:34.804399 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:34.804400 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:34.804401 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:34.804413 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:34.805917 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:34.806333 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:34.806334 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:34.806334 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:34.806395 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:34.806667 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:34.806668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:34.806668 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:34.806789 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:34.806864 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:34.807169 (+ 305us) server_negotiation.cc:299] Negotiation successful
0928 06:31:34.807258 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 06:31:34.807585 6545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:34.797522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:39765 (local address 127.22.225.129:40353)
0928 06:31:34.797700 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:34.797743 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:34.798386 (+ 643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:34.798703 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:34.798709 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:34.799572 (+ 863us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:34.799879 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:34.799887 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:34.800874 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:34.800875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:34.801684 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:34.801687 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:34.802127 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:34.802128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:34.802258 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:34.802814 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:34.802822 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:34.804281 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:34.806002 (+ 1721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:34.806010 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:34.806013 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:34.806244 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:34.806466 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:34.806467 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:34.806467 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:34.806591 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:34.806854 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:34.806861 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:34.807023 (+ 162us) client_negotiation.cc:769] Sending connection context
0928 06:31:34.807135 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 06:31:34.807306 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":90,"threads_started":1}
WARNING: no policy specified for kudu/127.22.225.130@KRBTEST.COM; defaulting to no policy
I20260928 06:31:34.808298 6543 heartbeater.cc:347] Connected to a master server at 127.22.225.190:39765
I20260928 06:31:34.808424 6543 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:34.808666 6543 heartbeater.cc:511] Master 127.22.225.190:39765 requested a full tablet report, sending...
I20260928 06:31:34.809321 6352 ts_manager.cc:194] Registered new tserver with Master: 43a313fadad1418cbcebf59dd0ecb76a (127.22.225.129:34637)
I20260928 06:31:34.810009 6352 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.225.129@KRBTEST.COM'} at 127.22.225.129:40353
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab.
Entry for principal kudu/127.22.225.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab.
I20260928 06:31:34.828305 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.130:0
--local_ip_for_outbound_sockets=127.22.225.130
--webserver_interface=127.22.225.130
--webserver_port=0
--tserver_master_addrs=127.22.225.190:39765
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab
--principal=kudu/127.22.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:38327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:31:34.919359 6548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:34.919637 6548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:34.919690 6548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:34.921012 6548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:34.921084 6548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:34.921135 6548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.130
I20260928 06:31:34.922816 6548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:38327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab
--principal=kudu/127.22.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.225.130
--webserver_port=0
--tserver_master_addrs=127.22.225.190:39765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.130
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:31:34.923118 6548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:34.923372 6548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:34.928548 6556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:34.928542 6554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:34.928704 6553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:34.929008 6548 server_base.cc:1034] running on GCE node
I20260928 06:31:34.929239 6548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:34.929538 6548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:34.930732 6548 hybrid_clock.cc:648] HybridClock initialized: now 1790577094930705 us; error 42 us; skew 500 ppm
Sep 28 06:31:34 dist-test-slave-d59m krb5kdc[6307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577094, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:34.933877 6548 init.cc:377] Logged in from keytab as kudu/127.22.225.130@KRBTEST.COM (short username kudu)
I20260928 06:31:34.934427 6548 webserver.cc:460] Webserver started at http://127.22.225.130:46691/ using document root <none> and password file <none>
I20260928 06:31:34.934698 6548 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:34.934769 6548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:34.934929 6548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:34.936120 6548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-1/data/instance:
uuid: "116591f9d0bd4e1590c1d52410720989"
format_stamp: "Formatted at 2026-09-28 06:31:34 on dist-test-slave-d59m"
server_key: "3cc391bbcd488f873a445eefea2ad995"
server_key_iv: "1275b3bfdfbc1ae5d3d83e3b4647f947"
server_key_version: "encryptionkey@0"
I20260928 06:31:34.936496 6548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-1/wal/instance:
uuid: "116591f9d0bd4e1590c1d52410720989"
format_stamp: "Formatted at 2026-09-28 06:31:34 on dist-test-slave-d59m"
server_key: "3cc391bbcd488f873a445eefea2ad995"
server_key_iv: "1275b3bfdfbc1ae5d3d83e3b4647f947"
server_key_version: "encryptionkey@0"
I20260928 06:31:34.937912 6548 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:31:34.938802 6563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:34.939002 6548 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:34.939113 6548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-1/wal
uuid: "116591f9d0bd4e1590c1d52410720989"
format_stamp: "Formatted at 2026-09-28 06:31:34 on dist-test-slave-d59m"
server_key: "3cc391bbcd488f873a445eefea2ad995"
server_key_iv: "1275b3bfdfbc1ae5d3d83e3b4647f947"
server_key_version: "encryptionkey@0"
I20260928 06:31:34.939201 6548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:34.948231 6548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:34.950721 6548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:34.950878 6548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:34.951125 6548 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:31:34.951452 6548 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:31:34.951508 6548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:34.951552 6548 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:31:34.951579 6548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:34.957773 6548 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.130:35187
I20260928 06:31:34.957852 6676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.130:35187 every 8 connection(s)
I20260928 06:31:34.958151 6548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
I20260928 06:31:34.964093 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 6548
I20260928 06:31:34.964223 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-1/wal/instance
I20260928 06:31:34.964452 23430 external_mini_cluster.cc:1428] Setting key 16e9bb91e762a5ad106e74c5c000f3bf
Sep 28 06:31:34 dist-test-slave-d59m krb5kdc[6307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577094, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.130@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:31:34.970243 6391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:34.958925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:57725 (local address 127.22.225.190:39765)
0928 06:31:34.959001 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:34.959004 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:34.959298 (+ 294us) server_negotiation.cc:407] Connection header received
0928 06:31:34.960033 (+ 735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:34.960034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:34.960044 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:34.960089 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:34.961230 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:34.961851 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:34.963529 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:34.963741 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:34.963810 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:34.966714 (+ 2904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:34.966717 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:34.966717 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:34.966726 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:34.968600 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:34.969084 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:34.969085 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:34.969086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:34.969156 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:34.969473 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:34.969474 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:34.969474 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:34.969634 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:34.969713 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:34.970068 (+ 355us) server_negotiation.cc:299] Negotiation successful
0928 06:31:34.970152 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 06:31:34.970453 6679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:34.958901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:39765 (local address 127.22.225.130:57725)
0928 06:31:34.959117 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:34.959175 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:34.959926 (+ 751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:34.960172 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:34.960176 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:34.960931 (+ 755us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:34.961127 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:34.961132 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:34.961959 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:34.961960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:34.963417 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:34.963421 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:34.963837 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:34.963838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:34.963991 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:34.964663 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:34.964674 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:34.966593 (+ 1919us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:34.968675 (+ 2082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:34.968683 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:34.968685 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:34.968996 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:34.969247 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:34.969248 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:34.969248 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:34.969397 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:34.969719 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:34.969726 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:34.969910 (+ 184us) client_negotiation.cc:769] Sending connection context
0928 06:31:34.970029 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 06:31:34.970209 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":103,"threads_started":1}
I20260928 06:31:34.970945 6677 heartbeater.cc:347] Connected to a master server at 127.22.225.190:39765
I20260928 06:31:34.971060 6677 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:34.971276 6677 heartbeater.cc:511] Master 127.22.225.190:39765 requested a full tablet report, sending...
I20260928 06:31:34.971706 6352 ts_manager.cc:194] Registered new tserver with Master: 116591f9d0bd4e1590c1d52410720989 (127.22.225.130:35187)
I20260928 06:31:34.972105 6352 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.225.130@KRBTEST.COM'} at 127.22.225.130:57725
WARNING: no policy specified for kudu/127.22.225.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab.
Entry for principal kudu/127.22.225.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab.
I20260928 06:31:34.991684 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.131:0
--local_ip_for_outbound_sockets=127.22.225.131
--webserver_interface=127.22.225.131
--webserver_port=0
--tserver_master_addrs=127.22.225.190:39765
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab
--principal=kudu/127.22.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:38327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:31:35.079638 6682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:35.079852 6682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:35.079878 6682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:35.081137 6682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:35.081182 6682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:35.081210 6682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.131
I20260928 06:31:35.083191 6682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:38327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab
--principal=kudu/127.22.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.225.131
--webserver_port=0
--tserver_master_addrs=127.22.225.190:39765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.131
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:31:35.083491 6682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:35.083750 6682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:35.089102 6690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:35.089139 6687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:35.089110 6688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:35.089287 6682 server_base.cc:1034] running on GCE node
I20260928 06:31:35.089658 6682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:35.089915 6682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:35.091082 6682 hybrid_clock.cc:648] HybridClock initialized: now 1790577095091058 us; error 41 us; skew 500 ppm
Sep 28 06:31:35 dist-test-slave-d59m krb5kdc[6307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577095, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:35.093907 6682 init.cc:377] Logged in from keytab as kudu/127.22.225.131@KRBTEST.COM (short username kudu)
I20260928 06:31:35.094426 6682 webserver.cc:460] Webserver started at http://127.22.225.131:35703/ using document root <none> and password file <none>
I20260928 06:31:35.094687 6682 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:35.094758 6682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:35.094920 6682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:35.096019 6682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-2/data/instance:
uuid: "80789f481b4948bd9e8732283ec4db11"
format_stamp: "Formatted at 2026-09-28 06:31:35 on dist-test-slave-d59m"
server_key: "ea0c4a6e5e4502802908171df0bbb3e0"
server_key_iv: "65a6afeeee22a63c5edae87189c89117"
server_key_version: "encryptionkey@0"
I20260928 06:31:35.096410 6682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-2/wal/instance:
uuid: "80789f481b4948bd9e8732283ec4db11"
format_stamp: "Formatted at 2026-09-28 06:31:35 on dist-test-slave-d59m"
server_key: "ea0c4a6e5e4502802908171df0bbb3e0"
server_key_iv: "65a6afeeee22a63c5edae87189c89117"
server_key_version: "encryptionkey@0"
I20260928 06:31:35.097909 6682 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:35.098723 6697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:35.098910 6682 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:35.099026 6682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-2/wal
uuid: "80789f481b4948bd9e8732283ec4db11"
format_stamp: "Formatted at 2026-09-28 06:31:35 on dist-test-slave-d59m"
server_key: "ea0c4a6e5e4502802908171df0bbb3e0"
server_key_iv: "65a6afeeee22a63c5edae87189c89117"
server_key_version: "encryptionkey@0"
I20260928 06:31:35.099112 6682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:35.139324 6682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:35.141644 6682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:35.141815 6682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:35.142079 6682 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:31:35.142434 6682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:31:35.142724 6682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:35.142810 6682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:31:35.142836 6682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:35.150656 6682 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.131:46263
I20260928 06:31:35.150739 6810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.131:46263 every 8 connection(s)
I20260928 06:31:35.151103 6682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
I20260928 06:31:35.157454 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 6682
I20260928 06:31:35.157562 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-2/wal/instance
I20260928 06:31:35.157828 23430 external_mini_cluster.cc:1428] Setting key c0266044746f28aa03223d37da9199ca
Sep 28 06:31:35 dist-test-slave-d59m krb5kdc[6307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577095, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.131@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:31:35.161799 6391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:35.151923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:51095 (local address 127.22.225.190:39765)
0928 06:31:35.151997 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:35.151999 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:35.152176 (+ 177us) server_negotiation.cc:407] Connection header received
0928 06:31:35.152801 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:35.152803 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:35.152815 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:35.152903 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:35.153873 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:35.154504 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:35.155581 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:35.155823 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:35.155899 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:35.158672 (+ 2773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:35.158674 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:35.158674 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:35.158683 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:35.160306 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:35.160829 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:35.160830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:35.160830 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:35.160897 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:35.161152 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:35.161153 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:35.161153 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:35.161304 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:35.161370 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:35.161635 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 06:31:35.161717 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 06:31:35.161921 6813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:35.151872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:39765 (local address 127.22.225.131:51095)
0928 06:31:35.152048 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:35.152094 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:35.152684 (+ 590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:35.152960 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:35.152964 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:35.153561 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:35.153766 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:35.153770 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:35.154633 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:35.154634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:35.155484 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:35.155487 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:35.155919 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:35.155920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:35.156034 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:35.156686 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:35.156695 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:35.158512 (+ 1817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:35.160407 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:35.160414 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:35.160416 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:35.160756 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:35.160971 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:35.160971 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:35.160971 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:35.161078 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:35.161369 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:35.161375 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:35.161499 (+ 124us) client_negotiation.cc:769] Sending connection context
0928 06:31:35.161578 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 06:31:35.161703 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":100,"threads_started":1}
I20260928 06:31:35.162438 6811 heartbeater.cc:347] Connected to a master server at 127.22.225.190:39765
I20260928 06:31:35.162564 6811 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:35.162755 6811 heartbeater.cc:511] Master 127.22.225.190:39765 requested a full tablet report, sending...
I20260928 06:31:35.163177 6352 ts_manager.cc:194] Registered new tserver with Master: 80789f481b4948bd9e8732283ec4db11 (127.22.225.131:46263)
I20260928 06:31:35.163556 6352 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.225.131@KRBTEST.COM'} at 127.22.225.131:51095
I20260928 06:31:35.170321 23430 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:31:35.178618 6391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:35.171919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47206 (local address 127.22.225.190:39765)
0928 06:31:35.171979 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:35.171981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:35.172077 (+ 96us) server_negotiation.cc:407] Connection header received
0928 06:31:35.172142 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:35.172143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:35.172154 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:35.172208 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:35.172833 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:35.173458 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:35.174185 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:35.174420 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:35.174493 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:35.175766 (+ 1273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:35.175767 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:35.175767 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:35.175775 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:35.177452 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:35.177794 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:35.177795 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:35.177795 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:35.177862 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:35.178055 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:35.178056 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:35.178056 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:35.178262 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:35.178370 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:35.178454 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 06:31:35.178532 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 06:31:35.180720 6352 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47206:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:31:35.182664 6352 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:31:35.190377 6824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:35.185358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.131:46263 (local address 127.0.0.1:46488)
0928 06:31:35.185994 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:35.186005 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:35.186041 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:35.186818 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:35.186819 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:35.186821 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:35.186855 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:35.186855 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:35.187969 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:35.187970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:35.189136 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:35.189140 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:35.190085 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:35.190087 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:35.190158 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:35.190173 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 06:31:35.190237 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 06:31:35.190294 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":137,"threads_started":1}
I20260928 06:31:35.190681 6827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:35.185707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47248 (local address 127.22.225.129:34637)
0928 06:31:35.185895 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:35.185900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:35.185912 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:31:35.185957 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:35.185959 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:35.185977 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:35.186061 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:35.186323 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:35.187514 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:35.189252 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:35.190001 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:35.190075 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:35.190129 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:35.190431 (+ 302us) server_negotiation.cc:299] Negotiation successful
0928 06:31:35.190593 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":105,"threads_started":1}
I20260928 06:31:35.190851 6823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:35.185342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.130:35187 (local address 127.0.0.1:34874)
0928 06:31:35.185832 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:35.185845 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:35.185916 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:35.186379 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:35.186380 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:35.186382 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:35.186436 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:35.186437 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:35.187743 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:35.187744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:35.189355 (+ 1611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:35.189356 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:35.190591 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:35.190592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:35.190657 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:35.190662 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:31:35.190695 (+ 33us) client_negotiation.cc:240] Negotiation successful
0928 06:31:35.190737 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"mutex_wait_us":55,"thread_start_us":102,"threads_started":1}
I20260928 06:31:35.190934 6822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:35.185342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.129:34637 (local address 127.0.0.1:47248)
0928 06:31:35.185486 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:35.185550 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:35.185628 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:35.186156 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:35.186159 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:35.186161 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:35.186238 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:35.186240 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:35.187642 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:35.187643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:35.189155 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:35.189158 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:35.190168 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:35.190170 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:35.190241 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:35.190246 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:31:35.190297 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 06:31:35.190349 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":43,"thread_start_us":80,"threads_started":1}
I20260928 06:31:35.191291 6825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:35.185647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34874 (local address 127.22.225.130:35187)
0928 06:31:35.185952 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:35.186143 (+ 191us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:35.186157 (+ 14us) server_negotiation.cc:407] Connection header received
0928 06:31:35.186202 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:35.186204 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:35.186220 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:35.186292 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:35.186529 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:35.187634 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:35.189445 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:35.190171 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:35.191054 (+ 883us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:35.191124 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:35.191169 (+ 45us) server_negotiation.cc:299] Negotiation successful
0928 06:31:35.191212 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":90,"threads_started":1}
I20260928 06:31:35.191413 6826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:35.185659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46488 (local address 127.22.225.131:46263)
0928 06:31:35.186586 (+ 927us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:35.186591 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:35.186601 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:31:35.186641 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:35.186643 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:35.186657 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:35.186724 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:35.186927 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:35.187845 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:35.189241 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:35.189917 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:35.189990 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:35.190049 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:35.191288 (+ 1239us) server_negotiation.cc:299] Negotiation successful
0928 06:31:35.191335 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":885,"thread_start_us":82,"threads_started":1}
I20260928 06:31:35.191711 6477 tablet_service.cc:1467] Processing CreateTablet for tablet 8706ea27516c48a9a4e432a7be0742a5 (DEFAULT_TABLE table=test-table [id=7506e0c06c4149a7bb3731f2c443d0c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:31:35.192010 6477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8706ea27516c48a9a4e432a7be0742a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:35.191939 6611 tablet_service.cc:1467] Processing CreateTablet for tablet 8706ea27516c48a9a4e432a7be0742a5 (DEFAULT_TABLE table=test-table [id=7506e0c06c4149a7bb3731f2c443d0c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:31:35.192252 6611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8706ea27516c48a9a4e432a7be0742a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:35.193248 6745 tablet_service.cc:1467] Processing CreateTablet for tablet 8706ea27516c48a9a4e432a7be0742a5 (DEFAULT_TABLE table=test-table [id=7506e0c06c4149a7bb3731f2c443d0c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:31:35.193586 6745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8706ea27516c48a9a4e432a7be0742a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:35.195444 6828 tablet_bootstrap.cc:492] T 8706ea27516c48a9a4e432a7be0742a5 P 43a313fadad1418cbcebf59dd0ecb76a: Bootstrap starting.
I20260928 06:31:35.195535 6829 tablet_bootstrap.cc:492] T 8706ea27516c48a9a4e432a7be0742a5 P 116591f9d0bd4e1590c1d52410720989: Bootstrap starting.
I20260928 06:31:35.196287 6829 tablet_bootstrap.cc:654] T 8706ea27516c48a9a4e432a7be0742a5 P 116591f9d0bd4e1590c1d52410720989: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:35.196295 6828 tablet_bootstrap.cc:654] T 8706ea27516c48a9a4e432a7be0742a5 P 43a313fadad1418cbcebf59dd0ecb76a: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:35.196730 6830 tablet_bootstrap.cc:492] T 8706ea27516c48a9a4e432a7be0742a5 P 80789f481b4948bd9e8732283ec4db11: Bootstrap starting.
I20260928 06:31:35.196979 6828 log.cc:824] T 8706ea27516c48a9a4e432a7be0742a5 P 43a313fadad1418cbcebf59dd0ecb76a: Log is configured to *not* fsync() on all Append() calls
I20260928 06:31:35.197077 6829 log.cc:824] T 8706ea27516c48a9a4e432a7be0742a5 P 116591f9d0bd4e1590c1d52410720989: Log is configured to *not* fsync() on all Append() calls
I20260928 06:31:35.197466 6830 tablet_bootstrap.cc:654] T 8706ea27516c48a9a4e432a7be0742a5 P 80789f481b4948bd9e8732283ec4db11: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:35.197839 6829 tablet_bootstrap.cc:492] T 8706ea27516c48a9a4e432a7be0742a5 P 116591f9d0bd4e1590c1d52410720989: No bootstrap required, opened a new log
I20260928 06:31:35.197913 6829 ts_tablet_manager.cc:1397] T 8706ea27516c48a9a4e432a7be0742a5 P 116591f9d0bd4e1590c1d52410720989: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 06:31:35.198073 6830 log.cc:824] T 8706ea27516c48a9a4e432a7be0742a5 P 80789f481b4948bd9e8732283ec4db11: Log is configured to *not* fsync() on all Append() calls
I20260928 06:31:35.198832 6830 tablet_bootstrap.cc:492] T 8706ea27516c48a9a4e432a7be0742a5 P 80789f481b4948bd9e8732283ec4db11: No bootstrap required, opened a new log
I20260928 06:31:35.198928 6830 ts_tablet_manager.cc:1397] T 8706ea27516c48a9a4e432a7be0742a5 P 80789f481b4948bd9e8732283ec4db11: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:31:35.199743 6829 raft_consensus.cc:348] T 8706ea27516c48a9a4e432a7be0742a5 P 116591f9d0bd4e1590c1d52410720989 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "116591f9d0bd4e1590c1d52410720989" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 35187 } } peers { permanent_uuid: "80789f481b4948bd9e8732283ec4db11" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 46263 } } peers { permanent_uuid: "43a313fadad1418cbcebf59dd0ecb76a" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34637 } }
I20260928 06:31:35.199946 6829 raft_consensus.cc:374] T 8706ea27516c48a9a4e432a7be0742a5 P 116591f9d0bd4e1590c1d52410720989 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:35.199982 6829 raft_consensus.cc:729] T 8706ea27516c48a9a4e432a7be0742a5 P 116591f9d0bd4e1590c1d52410720989 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 116591f9d0bd4e1590c1d52410720989, State: Initialized, Role: FOLLOWER
I20260928 06:31:35.200153 6829 consensus_queue.cc:260] T 8706ea27516c48a9a4e432a7be0742a5 P 116591f9d0bd4e1590c1d52410720989 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "116591f9d0bd4e1590c1d52410720989" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 35187 } } peers { permanent_uuid: "80789f481b4948bd9e8732283ec4db11" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 46263 } } peers { permanent_uuid: "43a313fadad1418cbcebf59dd0ecb76a" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34637 } }
I20260928 06:31:35.200445 6829 ts_tablet_manager.cc:1428] T 8706ea27516c48a9a4e432a7be0742a5 P 116591f9d0bd4e1590c1d52410720989: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 06:31:35.200644 6677 heartbeater.cc:503] Master 127.22.225.190:39765 was elected leader, sending a full tablet report...
I20260928 06:31:35.200562 6830 raft_consensus.cc:348] T 8706ea27516c48a9a4e432a7be0742a5 P 80789f481b4948bd9e8732283ec4db11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "116591f9d0bd4e1590c1d52410720989" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 35187 } } peers { permanent_uuid: "80789f481b4948bd9e8732283ec4db11" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 46263 } } peers { permanent_uuid: "43a313fadad1418cbcebf59dd0ecb76a" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34637 } }
I20260928 06:31:35.200760 6830 raft_consensus.cc:374] T 8706ea27516c48a9a4e432a7be0742a5 P 80789f481b4948bd9e8732283ec4db11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:35.200795 6830 raft_consensus.cc:729] T 8706ea27516c48a9a4e432a7be0742a5 P 80789f481b4948bd9e8732283ec4db11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80789f481b4948bd9e8732283ec4db11, State: Initialized, Role: FOLLOWER
I20260928 06:31:35.200951 6828 tablet_bootstrap.cc:492] T 8706ea27516c48a9a4e432a7be0742a5 P 43a313fadad1418cbcebf59dd0ecb76a: No bootstrap required, opened a new log
I20260928 06:31:35.201013 6828 ts_tablet_manager.cc:1397] T 8706ea27516c48a9a4e432a7be0742a5 P 43a313fadad1418cbcebf59dd0ecb76a: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20260928 06:31:35.201064 6830 consensus_queue.cc:260] T 8706ea27516c48a9a4e432a7be0742a5 P 80789f481b4948bd9e8732283ec4db11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "116591f9d0bd4e1590c1d52410720989" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 35187 } } peers { permanent_uuid: "80789f481b4948bd9e8732283ec4db11" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 46263 } } peers { permanent_uuid: "43a313fadad1418cbcebf59dd0ecb76a" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34637 } }
I20260928 06:31:35.201465 6830 ts_tablet_manager.cc:1428] T 8706ea27516c48a9a4e432a7be0742a5 P 80789f481b4948bd9e8732283ec4db11: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 06:31:35.201584 6811 heartbeater.cc:503] Master 127.22.225.190:39765 was elected leader, sending a full tablet report...
I20260928 06:31:35.202883 6828 raft_consensus.cc:348] T 8706ea27516c48a9a4e432a7be0742a5 P 43a313fadad1418cbcebf59dd0ecb76a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "116591f9d0bd4e1590c1d52410720989" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 35187 } } peers { permanent_uuid: "80789f481b4948bd9e8732283ec4db11" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 46263 } } peers { permanent_uuid: "43a313fadad1418cbcebf59dd0ecb76a" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34637 } }
I20260928 06:31:35.203068 6828 raft_consensus.cc:374] T 8706ea27516c48a9a4e432a7be0742a5 P 43a313fadad1418cbcebf59dd0ecb76a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:35.203099 6828 raft_consensus.cc:729] T 8706ea27516c48a9a4e432a7be0742a5 P 43a313fadad1418cbcebf59dd0ecb76a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43a313fadad1418cbcebf59dd0ecb76a, State: Initialized, Role: FOLLOWER
I20260928 06:31:35.203207 6828 consensus_queue.cc:260] T 8706ea27516c48a9a4e432a7be0742a5 P 43a313fadad1418cbcebf59dd0ecb76a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "116591f9d0bd4e1590c1d52410720989" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 35187 } } peers { permanent_uuid: "80789f481b4948bd9e8732283ec4db11" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 46263 } } peers { permanent_uuid: "43a313fadad1418cbcebf59dd0ecb76a" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34637 } }
I20260928 06:31:35.203464 6828 ts_tablet_manager.cc:1428] T 8706ea27516c48a9a4e432a7be0742a5 P 43a313fadad1418cbcebf59dd0ecb76a: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260928 06:31:35.203495 6543 heartbeater.cc:503] Master 127.22.225.190:39765 was elected leader, sending a full tablet report...
W20260928 06:31:35.208775 6678 tablet.cc:2369] T 8706ea27516c48a9a4e432a7be0742a5 P 116591f9d0bd4e1590c1d52410720989: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:31:35.297567 6544 tablet.cc:2369] T 8706ea27516c48a9a4e432a7be0742a5 P 43a313fadad1418cbcebf59dd0ecb76a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:31:35.321802 6835 raft_consensus.cc:482] T 8706ea27516c48a9a4e432a7be0742a5 P 80789f481b4948bd9e8732283ec4db11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:31:35.321972 6835 raft_consensus.cc:504] T 8706ea27516c48a9a4e432a7be0742a5 P 80789f481b4948bd9e8732283ec4db11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "116591f9d0bd4e1590c1d52410720989" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 35187 } } peers { permanent_uuid: "80789f481b4948bd9e8732283ec4db11" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 46263 } } peers { permanent_uuid: "43a313fadad1418cbcebf59dd0ecb76a" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34637 } }
I20260928 06:31:35.322392 6835 leader_election.cc:288] T 8706ea27516c48a9a4e432a7be0742a5 P 80789f481b4948bd9e8732283ec4db11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 116591f9d0bd4e1590c1d52410720989 (127.22.225.130:35187), 43a313fadad1418cbcebf59dd0ecb76a (127.22.225.129:34637)
I20260928 06:31:35.326054 6827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:35.322841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:49687 (local address 127.22.225.129:34637)
0928 06:31:35.322891 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:35.322893 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:35.322917 (+ 24us) server_negotiation.cc:407] Connection header received
0928 06:31:35.322998 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:35.323000 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:35.323011 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:35.323043 (+ 32us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:35.323291 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:35.323921 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:35.325237 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:35.325637 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:35.325679 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:35.325686 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:35.325933 (+ 247us) server_negotiation.cc:299] Negotiation successful
0928 06:31:35.325980 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 06:31:35.326079 6825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:35.322753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:43161 (local address 127.22.225.130:35187)
0928 06:31:35.322811 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:35.322813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:35.322824 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:31:35.322858 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:35.322860 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:35.322871 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:35.322946 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:35.323173 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:35.323690 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:35.325175 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:35.325689 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:35.325776 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:35.325792 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:35.325973 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 06:31:35.326018 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 06:31:35.326076 6813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:35.322621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.130:35187 (local address 127.22.225.131:43161)
0928 06:31:35.322677 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:35.322690 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:35.322739 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:35.323006 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:35.323007 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:35.323009 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:35.323083 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:35.323085 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:35.323784 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:35.323785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:35.325053 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:35.325057 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:35.325784 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:35.325785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:35.325866 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:35.325869 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:31:35.325920 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 06:31:35.325996 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":3}
I20260928 06:31:35.326448 6837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:35.322622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.129:34637 (local address 127.22.225.131:49687)
0928 06:31:35.322859 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:35.322868 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:35.322912 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:35.323102 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:35.323102 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:35.323103 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:35.323198 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:35.323200 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:35.324015 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:35.324015 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:35.325137 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:35.325139 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:35.325765 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:35.325765 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:35.325848 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:35.325852 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:31:35.325920 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 06:31:35.326377 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":466,"thread_start_us":106,"threads_started":1}
I20260928 06:31:35.326437 6631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8706ea27516c48a9a4e432a7be0742a5" candidate_uuid: "80789f481b4948bd9e8732283ec4db11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "116591f9d0bd4e1590c1d52410720989" is_pre_election: true
I20260928 06:31:35.326601 6631 raft_consensus.cc:2454] T 8706ea27516c48a9a4e432a7be0742a5 P 116591f9d0bd4e1590c1d52410720989 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80789f481b4948bd9e8732283ec4db11 in term 0.
I20260928 06:31:35.326866 6698 leader_election.cc:302] T 8706ea27516c48a9a4e432a7be0742a5 P 80789f481b4948bd9e8732283ec4db11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 116591f9d0bd4e1590c1d52410720989, 80789f481b4948bd9e8732283ec4db11; no voters:
I20260928 06:31:35.326900 6497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8706ea27516c48a9a4e432a7be0742a5" candidate_uuid: "80789f481b4948bd9e8732283ec4db11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43a313fadad1418cbcebf59dd0ecb76a" is_pre_election: true
I20260928 06:31:35.327029 6497 raft_consensus.cc:2454] T 8706ea27516c48a9a4e432a7be0742a5 P 43a313fadad1418cbcebf59dd0ecb76a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80789f481b4948bd9e8732283ec4db11 in term 0.
I20260928 06:31:35.327029 6835 raft_consensus.cc:2781] T 8706ea27516c48a9a4e432a7be0742a5 P 80789f481b4948bd9e8732283ec4db11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:31:35.327082 6835 raft_consensus.cc:482] T 8706ea27516c48a9a4e432a7be0742a5 P 80789f481b4948bd9e8732283ec4db11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:31:35.327116 6835 raft_consensus.cc:3037] T 8706ea27516c48a9a4e432a7be0742a5 P 80789f481b4948bd9e8732283ec4db11 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:35.327917 6835 raft_consensus.cc:504] T 8706ea27516c48a9a4e432a7be0742a5 P 80789f481b4948bd9e8732283ec4db11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "116591f9d0bd4e1590c1d52410720989" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 35187 } } peers { permanent_uuid: "80789f481b4948bd9e8732283ec4db11" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 46263 } } peers { permanent_uuid: "43a313fadad1418cbcebf59dd0ecb76a" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34637 } }
I20260928 06:31:35.328153 6835 leader_election.cc:288] T 8706ea27516c48a9a4e432a7be0742a5 P 80789f481b4948bd9e8732283ec4db11 [CANDIDATE]: Term 1 election: Requested vote from peers 116591f9d0bd4e1590c1d52410720989 (127.22.225.130:35187), 43a313fadad1418cbcebf59dd0ecb76a (127.22.225.129:34637)
I20260928 06:31:35.328305 6631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8706ea27516c48a9a4e432a7be0742a5" candidate_uuid: "80789f481b4948bd9e8732283ec4db11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "116591f9d0bd4e1590c1d52410720989"
I20260928 06:31:35.328322 6497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8706ea27516c48a9a4e432a7be0742a5" candidate_uuid: "80789f481b4948bd9e8732283ec4db11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43a313fadad1418cbcebf59dd0ecb76a"
I20260928 06:31:35.328414 6497 raft_consensus.cc:3037] T 8706ea27516c48a9a4e432a7be0742a5 P 43a313fadad1418cbcebf59dd0ecb76a [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:35.328404 6631 raft_consensus.cc:3037] T 8706ea27516c48a9a4e432a7be0742a5 P 116591f9d0bd4e1590c1d52410720989 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:35.329413 6497 raft_consensus.cc:2454] T 8706ea27516c48a9a4e432a7be0742a5 P 43a313fadad1418cbcebf59dd0ecb76a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80789f481b4948bd9e8732283ec4db11 in term 1.
I20260928 06:31:35.329406 6631 raft_consensus.cc:2454] T 8706ea27516c48a9a4e432a7be0742a5 P 116591f9d0bd4e1590c1d52410720989 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80789f481b4948bd9e8732283ec4db11 in term 1.
I20260928 06:31:35.329625 6701 leader_election.cc:302] T 8706ea27516c48a9a4e432a7be0742a5 P 80789f481b4948bd9e8732283ec4db11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43a313fadad1418cbcebf59dd0ecb76a, 80789f481b4948bd9e8732283ec4db11; no voters:
I20260928 06:31:35.329710 6835 raft_consensus.cc:2781] T 8706ea27516c48a9a4e432a7be0742a5 P 80789f481b4948bd9e8732283ec4db11 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:31:35.329864 6835 raft_consensus.cc:686] T 8706ea27516c48a9a4e432a7be0742a5 P 80789f481b4948bd9e8732283ec4db11 [term 1 LEADER]: Becoming Leader. State: Replica: 80789f481b4948bd9e8732283ec4db11, State: Running, Role: LEADER
I20260928 06:31:35.330001 6835 consensus_queue.cc:237] T 8706ea27516c48a9a4e432a7be0742a5 P 80789f481b4948bd9e8732283ec4db11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "116591f9d0bd4e1590c1d52410720989" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 35187 } } peers { permanent_uuid: "80789f481b4948bd9e8732283ec4db11" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 46263 } } peers { permanent_uuid: "43a313fadad1418cbcebf59dd0ecb76a" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34637 } }
I20260928 06:31:35.331048 6351 catalog_manager.cc:5495] T 8706ea27516c48a9a4e432a7be0742a5 P 80789f481b4948bd9e8732283ec4db11 reported cstate change: term changed from 0 to 1, leader changed from <none> to 80789f481b4948bd9e8732283ec4db11 (127.22.225.131). New cstate: current_term: 1 leader_uuid: "80789f481b4948bd9e8732283ec4db11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "116591f9d0bd4e1590c1d52410720989" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 35187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80789f481b4948bd9e8732283ec4db11" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 46263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43a313fadad1418cbcebf59dd0ecb76a" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34637 } health_report { overall_health: UNKNOWN } } }
I20260928 06:31:35.338672 6826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:35.335991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37634 (local address 127.22.225.131:46263)
0928 06:31:35.336063 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:35.336065 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:35.336075 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:31:35.336116 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:35.336117 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:35.336140 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:35.336223 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:31:35.336333 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:35.337036 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:35.337831 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:35.338096 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:35.338170 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:35.338310 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:31:35.338439 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:31:35.338523 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:35.338540 (+ 17us) server_negotiation.cc:299] Negotiation successful
0928 06:31:35.338562 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 06:31:35.340687 6497 raft_consensus.cc:1261] T 8706ea27516c48a9a4e432a7be0742a5 P 43a313fadad1418cbcebf59dd0ecb76a [term 1 FOLLOWER]: Refusing update from remote peer 80789f481b4948bd9e8732283ec4db11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:31:35.340705 6631 raft_consensus.cc:1261] T 8706ea27516c48a9a4e432a7be0742a5 P 116591f9d0bd4e1590c1d52410720989 [term 1 FOLLOWER]: Refusing update from remote peer 80789f481b4948bd9e8732283ec4db11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:31:35.341105 6838 consensus_queue.cc:1035] T 8706ea27516c48a9a4e432a7be0742a5 P 80789f481b4948bd9e8732283ec4db11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43a313fadad1418cbcebf59dd0ecb76a" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:31:35.341360 6835 consensus_queue.cc:1035] T 8706ea27516c48a9a4e432a7be0742a5 P 80789f481b4948bd9e8732283ec4db11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "116591f9d0bd4e1590c1d52410720989" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 35187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260928 06:31:35.347165 6351 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47206:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:31:35.347303 6351 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47206:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:31:35.347656 6845 mvcc.cc:205] Tried to move back new op lower bound from 7334203782511771648 to 7334203782471979008. Current Snapshot: MvccSnapshot[applied={T|T < 7334203782511771648}]
I20260928 06:31:35.348063 6351 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c3eacb4d322f415eb7bb4b021abfbece: Sending DeleteTablet for 3 replicas of tablet 8706ea27516c48a9a4e432a7be0742a5
I20260928 06:31:35.348419 6611 tablet_service.cc:1514] Processing DeleteTablet for tablet 8706ea27516c48a9a4e432a7be0742a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:31:35 UTC) from {username='kudu', principal='kudu/127.22.225.190@KRBTEST.COM'} at 127.0.0.1:34874
I20260928 06:31:35.348451 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 6414
I20260928 06:31:35.348565 6477 tablet_service.cc:1514] Processing DeleteTablet for tablet 8706ea27516c48a9a4e432a7be0742a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:31:35 UTC) from {username='kudu', principal='kudu/127.22.225.190@KRBTEST.COM'} at 127.0.0.1:47248
I20260928 06:31:35.356509 6745 tablet_service.cc:1514] Processing DeleteTablet for tablet 8706ea27516c48a9a4e432a7be0742a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:31:35 UTC) from {username='kudu', principal='kudu/127.22.225.190@KRBTEST.COM'} at 127.0.0.1:46488
I20260928 06:31:35.356858 6852 tablet_replica.cc:330] T 8706ea27516c48a9a4e432a7be0742a5 P 80789f481b4948bd9e8732283ec4db11: stopping tablet replica
I20260928 06:31:35.357007 6852 raft_consensus.cc:2229] T 8706ea27516c48a9a4e432a7be0742a5 P 80789f481b4948bd9e8732283ec4db11 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:31:35.357163 6852 raft_consensus.cc:2258] T 8706ea27516c48a9a4e432a7be0742a5 P 80789f481b4948bd9e8732283ec4db11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:31:35.357795 6852 ts_tablet_manager.cc:1905] T 8706ea27516c48a9a4e432a7be0742a5 P 80789f481b4948bd9e8732283ec4db11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:31:35.359907 6852 ts_tablet_manager.cc:1918] T 8706ea27516c48a9a4e432a7be0742a5 P 80789f481b4948bd9e8732283ec4db11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:31:35.359982 6852 log.cc:1196] T 8706ea27516c48a9a4e432a7be0742a5 P 80789f481b4948bd9e8732283ec4db11: Deleting WAL directory at /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-2/wal/wals/8706ea27516c48a9a4e432a7be0742a5
I20260928 06:31:35.360248 6852 ts_tablet_manager.cc:1939] T 8706ea27516c48a9a4e432a7be0742a5 P 80789f481b4948bd9e8732283ec4db11: Deleting consensus metadata
I20260928 06:31:35.360747 6338 catalog_manager.cc:4841] TS 80789f481b4948bd9e8732283ec4db11 (127.22.225.131:46263): tablet 8706ea27516c48a9a4e432a7be0742a5 (table test-table [id=7506e0c06c4149a7bb3731f2c443d0c0]) successfully deleted
I20260928 06:31:35.360956 6851 tablet_replica.cc:330] T 8706ea27516c48a9a4e432a7be0742a5 P 116591f9d0bd4e1590c1d52410720989: stopping tablet replica
I20260928 06:31:35.361088 6851 raft_consensus.cc:2229] T 8706ea27516c48a9a4e432a7be0742a5 P 116591f9d0bd4e1590c1d52410720989 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:31:35.361212 6851 raft_consensus.cc:2258] T 8706ea27516c48a9a4e432a7be0742a5 P 116591f9d0bd4e1590c1d52410720989 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:31:35.361766 6851 ts_tablet_manager.cc:1905] T 8706ea27516c48a9a4e432a7be0742a5 P 116591f9d0bd4e1590c1d52410720989: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 06:31:35.362660 6339 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.225.129:34637 (error 108)
I20260928 06:31:35.363721 6851 ts_tablet_manager.cc:1918] T 8706ea27516c48a9a4e432a7be0742a5 P 116591f9d0bd4e1590c1d52410720989: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:31:35.363811 6851 log.cc:1196] T 8706ea27516c48a9a4e432a7be0742a5 P 116591f9d0bd4e1590c1d52410720989: Deleting WAL directory at /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-1/wal/wals/8706ea27516c48a9a4e432a7be0742a5
I20260928 06:31:35.364095 6851 ts_tablet_manager.cc:1939] T 8706ea27516c48a9a4e432a7be0742a5 P 116591f9d0bd4e1590c1d52410720989: Deleting consensus metadata
I20260928 06:31:35.364815 6336 catalog_manager.cc:4841] TS 116591f9d0bd4e1590c1d52410720989 (127.22.225.130:35187): tablet 8706ea27516c48a9a4e432a7be0742a5 (table test-table [id=7506e0c06c4149a7bb3731f2c443d0c0]) successfully deleted
W20260928 06:31:35.371279 6824 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:31:35.363253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.129:34637 (local address 127.0.0.1:60206)
0928 06:31:35.363308 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:35.363322 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:35.363379 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:35.371212 (+ 7833us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.129:34637: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 06:31:35.371479 6339 catalog_manager.cc:4584] TS 43a313fadad1418cbcebf59dd0ecb76a (127.22.225.129:34637): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8706ea27516c48a9a4e432a7be0742a5: Network error: Client connection negotiation failed: client connection to 127.22.225.129:34637: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 06:31:35.371934 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 6548
I20260928 06:31:35.379662 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 6682
I20260928 06:31:35.388162 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 6320
I20260928 06:31:55.396862 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:39765
--webserver_interface=127.22.225.190
--webserver_port=45977
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab
--principal=kudu/127.22.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:38327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.225.190:39765
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:31:55.481746 6854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:55.481973 6854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:55.481997 6854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:55.483158 6854 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 06:31:55.483198 6854 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 06:31:55.483211 6854 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 06:31:55.483223 6854 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:31:55.483240 6854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:55.483253 6854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:55.483309 6854 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:31:55.483325 6854 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:31:55.484880 6854 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:38327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.22.225.190:39765
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab
--principal=kudu/127.22.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:39765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.225.190
--webserver_port=45977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:31:55.485121 6854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:55.485381 6854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:55.496258 6861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:55.496241 6863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:55.496568 6854 server_base.cc:1034] running on GCE node
W20260928 06:31:55.496248 6860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:55.496987 6854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:55.497385 6854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:55.498570 6854 hybrid_clock.cc:648] HybridClock initialized: now 1790577115498546 us; error 42 us; skew 500 ppm
Sep 28 06:31:55 dist-test-slave-d59m krb5kdc[6307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577115, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:55.505157 6854 init.cc:377] Logged in from keytab as kudu/127.22.225.190@KRBTEST.COM (short username kudu)
I20260928 06:31:55.505781 6854 webserver.cc:460] Webserver started at http://127.22.225.190:45977/ using document root <none> and password file <none>
I20260928 06:31:55.506083 6854 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:55.506150 6854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:55.507786 6854 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:55.508617 6870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:55.508802 6854 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:55.508931 6854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/master-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/master-0/wal
uuid: "c3eacb4d322f415eb7bb4b021abfbece"
format_stamp: "Formatted at 2026-09-28 06:31:34 on dist-test-slave-d59m"
server_key: "0ae844b205141afacca93022773b09dd"
server_key_iv: "88b2d78127515b9607f76597944b916c"
server_key_version: "encryptionkey@0"
I20260928 06:31:55.509403 6854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:55.524103 6854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:55.526654 6854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:55.526859 6854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:55.532327 6854 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.190:39765
I20260928 06:31:55.532359 6922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.190:39765 every 8 connection(s)
I20260928 06:31:55.532884 6854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
I20260928 06:31:55.534564 6923 sys_catalog.cc:263] Verifying existing consensus state
I20260928 06:31:55.535320 6923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3eacb4d322f415eb7bb4b021abfbece: Bootstrap starting.
I20260928 06:31:55.542429 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 6854
I20260928 06:31:55.542773 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.129:34637
--local_ip_for_outbound_sockets=127.22.225.129
--tserver_master_addrs=127.22.225.190:39765
--webserver_port=43921
--webserver_interface=127.22.225.129
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab
--principal=kudu/127.22.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:38327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:31:55.545996 6923 log.cc:824] T 00000000000000000000000000000000 P c3eacb4d322f415eb7bb4b021abfbece: Log is configured to *not* fsync() on all Append() calls
I20260928 06:31:55.549363 6923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3eacb4d322f415eb7bb4b021abfbece: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:31:55.549830 6923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3eacb4d322f415eb7bb4b021abfbece: Bootstrap complete.
I20260928 06:31:55.554543 6923 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3eacb4d322f415eb7bb4b021abfbece [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3eacb4d322f415eb7bb4b021abfbece" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 39765 } }
I20260928 06:31:55.555013 6923 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3eacb4d322f415eb7bb4b021abfbece [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3eacb4d322f415eb7bb4b021abfbece, State: Initialized, Role: FOLLOWER
I20260928 06:31:55.555289 6923 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3eacb4d322f415eb7bb4b021abfbece [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3eacb4d322f415eb7bb4b021abfbece" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 39765 } }
I20260928 06:31:55.555397 6923 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3eacb4d322f415eb7bb4b021abfbece [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:31:55.555519 6923 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3eacb4d322f415eb7bb4b021abfbece [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:31:55.555595 6923 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3eacb4d322f415eb7bb4b021abfbece [term 1 FOLLOWER]: Advancing to term 2
I20260928 06:31:55.557302 6923 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3eacb4d322f415eb7bb4b021abfbece [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3eacb4d322f415eb7bb4b021abfbece" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 39765 } }
I20260928 06:31:55.557425 6923 leader_election.cc:302] T 00000000000000000000000000000000 P c3eacb4d322f415eb7bb4b021abfbece [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3eacb4d322f415eb7bb4b021abfbece; no voters:
I20260928 06:31:55.557611 6923 leader_election.cc:288] T 00000000000000000000000000000000 P c3eacb4d322f415eb7bb4b021abfbece [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 06:31:55.557689 6927 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3eacb4d322f415eb7bb4b021abfbece [term 2 FOLLOWER]: Leader election won for term 2
I20260928 06:31:55.557906 6927 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3eacb4d322f415eb7bb4b021abfbece [term 2 LEADER]: Becoming Leader. State: Replica: c3eacb4d322f415eb7bb4b021abfbece, State: Running, Role: LEADER
I20260928 06:31:55.557971 6923 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3eacb4d322f415eb7bb4b021abfbece [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:31:55.558040 6927 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3eacb4d322f415eb7bb4b021abfbece [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3eacb4d322f415eb7bb4b021abfbece" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 39765 } }
I20260928 06:31:55.558408 6927 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3eacb4d322f415eb7bb4b021abfbece [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c3eacb4d322f415eb7bb4b021abfbece" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3eacb4d322f415eb7bb4b021abfbece" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 39765 } } }
I20260928 06:31:55.558538 6927 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3eacb4d322f415eb7bb4b021abfbece [sys.catalog]: This master's current role is: LEADER
I20260928 06:31:55.558694 6927 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3eacb4d322f415eb7bb4b021abfbece [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3eacb4d322f415eb7bb4b021abfbece. Latest consensus state: current_term: 2 leader_uuid: "c3eacb4d322f415eb7bb4b021abfbece" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3eacb4d322f415eb7bb4b021abfbece" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 39765 } } }
I20260928 06:31:55.558799 6927 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3eacb4d322f415eb7bb4b021abfbece [sys.catalog]: This master's current role is: LEADER
I20260928 06:31:55.558929 6934 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:31:55.559690 6934 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:31:55.559839 6934 catalog_manager.cc:1259] Loaded cluster ID: ec1aca84d3834c95a23cf3dea47cef2e
I20260928 06:31:55.560159 6934 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:31:55.561342 6934 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:31:55.561568 6934 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c3eacb4d322f415eb7bb4b021abfbece: Loaded TSK: 0
I20260928 06:31:55.573221 6934 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c3eacb4d322f415eb7bb4b021abfbece: Generated new TSK 1
I20260928 06:31:55.573311 6934 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 06:31:55.670676 6925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:55.670976 6925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:55.671020 6925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:55.672751 6925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:55.672819 6925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:55.672863 6925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.129
I20260928 06:31:55.675033 6925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:38327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab
--principal=kudu/127.22.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.129:34637
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.225.129
--webserver_port=43921
--tserver_master_addrs=127.22.225.190:39765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6925
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.129
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:31:55.675299 6925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:55.675570 6925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:55.681563 6950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:55.681605 6951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:55.681898 6953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:55.682514 6925 server_base.cc:1034] running on GCE node
I20260928 06:31:55.682713 6925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:55.682955 6925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:55.684110 6925 hybrid_clock.cc:648] HybridClock initialized: now 1790577115684093 us; error 32 us; skew 500 ppm
Sep 28 06:31:55 dist-test-slave-d59m krb5kdc[6307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577115, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:55.687327 6925 init.cc:377] Logged in from keytab as kudu/127.22.225.129@KRBTEST.COM (short username kudu)
I20260928 06:31:55.687870 6925 webserver.cc:460] Webserver started at http://127.22.225.129:43921/ using document root <none> and password file <none>
I20260928 06:31:55.688159 6925 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:55.688262 6925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:55.690186 6925 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:55.691190 6960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:55.691383 6925 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:55.691458 6925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-0/wal
uuid: "43a313fadad1418cbcebf59dd0ecb76a"
format_stamp: "Formatted at 2026-09-28 06:31:34 on dist-test-slave-d59m"
server_key: "ed44e41bb9338f5afaa97a8b586735cd"
server_key_iv: "ae625cd819ee12cd936dcbf37b7c62df"
server_key_version: "encryptionkey@0"
I20260928 06:31:55.691870 6925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:55.708851 6925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:55.711563 6925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:55.711707 6925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:55.711946 6925 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:31:55.712476 6967 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 06:31:55.714498 6925 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 06:31:55.714555 6925 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 06:31:55.714586 6925 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 06:31:55.715369 6925 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 06:31:55.715443 6925 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:55.715583 6967 tablet_bootstrap.cc:492] T 8706ea27516c48a9a4e432a7be0742a5 P 43a313fadad1418cbcebf59dd0ecb76a: Bootstrap starting.
I20260928 06:31:55.721928 6925 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.129:34637
I20260928 06:31:55.722389 6925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
I20260928 06:31:55.727506 6967 log.cc:824] T 8706ea27516c48a9a4e432a7be0742a5 P 43a313fadad1418cbcebf59dd0ecb76a: Log is configured to *not* fsync() on all Append() calls
I20260928 06:31:55.729647 7074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.129:34637 every 8 connection(s)
I20260928 06:31:55.729858 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 6925
I20260928 06:31:55.730151 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.130:35187
--local_ip_for_outbound_sockets=127.22.225.130
--tserver_master_addrs=127.22.225.190:39765
--webserver_port=46691
--webserver_interface=127.22.225.130
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab
--principal=kudu/127.22.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:38327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:31:55.732015 6967 tablet_bootstrap.cc:492] T 8706ea27516c48a9a4e432a7be0742a5 P 43a313fadad1418cbcebf59dd0ecb76a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:31:55.732383 6967 tablet_bootstrap.cc:492] T 8706ea27516c48a9a4e432a7be0742a5 P 43a313fadad1418cbcebf59dd0ecb76a: Bootstrap complete.
I20260928 06:31:55.733814 6967 ts_tablet_manager.cc:1397] T 8706ea27516c48a9a4e432a7be0742a5 P 43a313fadad1418cbcebf59dd0ecb76a: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.012s
I20260928 06:31:55.735145 6967 raft_consensus.cc:348] T 8706ea27516c48a9a4e432a7be0742a5 P 43a313fadad1418cbcebf59dd0ecb76a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "116591f9d0bd4e1590c1d52410720989" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 35187 } } peers { permanent_uuid: "80789f481b4948bd9e8732283ec4db11" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 46263 } } peers { permanent_uuid: "43a313fadad1418cbcebf59dd0ecb76a" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34637 } }
I20260928 06:31:55.735562 6967 raft_consensus.cc:729] T 8706ea27516c48a9a4e432a7be0742a5 P 43a313fadad1418cbcebf59dd0ecb76a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43a313fadad1418cbcebf59dd0ecb76a, State: Initialized, Role: FOLLOWER
I20260928 06:31:55.735751 6967 consensus_queue.cc:260] T 8706ea27516c48a9a4e432a7be0742a5 P 43a313fadad1418cbcebf59dd0ecb76a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "116591f9d0bd4e1590c1d52410720989" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 35187 } } peers { permanent_uuid: "80789f481b4948bd9e8732283ec4db11" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 46263 } } peers { permanent_uuid: "43a313fadad1418cbcebf59dd0ecb76a" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34637 } }
I20260928 06:31:55.736065 6967 ts_tablet_manager.cc:1428] T 8706ea27516c48a9a4e432a7be0742a5 P 43a313fadad1418cbcebf59dd0ecb76a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
Sep 28 06:31:55 dist-test-slave-d59m krb5kdc[6307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577115, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.129@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:31:55.744180 7078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:55.730433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:39765 (local address 127.22.225.129:47159)
0928 06:31:55.731650 (+ 1217us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:55.731699 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:55.732327 (+ 628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:55.733121 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:55.733126 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:55.733490 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:55.733793 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:55.733799 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:55.735507 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:55.735508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:55.736818 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:55.736821 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:55.737605 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:55.737606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:55.737750 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:55.738339 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:55.738349 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:55.740033 (+ 1684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:55.742263 (+ 2230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:55.742270 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:55.742272 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:55.742538 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:55.742806 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:55.742806 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:55.742807 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:55.742918 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:55.743349 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:55.743355 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:55.743592 (+ 237us) client_negotiation.cc:769] Sending connection context
0928 06:31:55.743781 (+ 189us) client_negotiation.cc:240] Negotiation successful
0928 06:31:55.743967 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1165,"thread_start_us":78,"threads_started":1}
I20260928 06:31:55.745278 7079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:55.730521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:47159 (local address 127.22.225.190:39765)
0928 06:31:55.732551 (+ 2030us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:55.732561 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:55.732635 (+ 74us) server_negotiation.cc:407] Connection header received
0928 06:31:55.732782 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:55.732789 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:55.732845 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:55.732977 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:55.733939 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:55.735373 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:55.736912 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:55.737504 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:55.737605 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:55.740169 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:55.740174 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:55.740175 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:55.740186 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:55.742159 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:55.742623 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:55.742626 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:55.742626 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:55.742736 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:55.742981 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:55.742982 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:55.742983 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:55.743253 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:55.743323 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:55.744924 (+ 1601us) server_negotiation.cc:299] Negotiation successful
0928 06:31:55.745058 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1973,"thread_start_us":101,"threads_started":1}
I20260928 06:31:55.746474 7075 heartbeater.cc:347] Connected to a master server at 127.22.225.190:39765
I20260928 06:31:55.746558 7075 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:55.746723 7075 heartbeater.cc:511] Master 127.22.225.190:39765 requested a full tablet report, sending...
I20260928 06:31:55.747452 6887 ts_manager.cc:194] Registered new tserver with Master: 43a313fadad1418cbcebf59dd0ecb76a (127.22.225.129:34637)
I20260928 06:31:55.748510 6887 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.225.129@KRBTEST.COM'} at 127.22.225.129:47159
I20260928 06:31:55.749495 7075 heartbeater.cc:503] Master 127.22.225.190:39765 was elected leader, sending a full tablet report...
I20260928 06:31:55.757164 7082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:55.748929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.129:34637 (local address 127.0.0.1:42538)
0928 06:31:55.752701 (+ 3772us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:55.752743 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:55.752813 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:55.753059 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:55.753062 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:55.753064 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:55.753120 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:55.753121 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:55.754244 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:55.754245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:55.755763 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:55.755767 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:55.756896 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:55.756897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:55.756974 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:55.756987 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 06:31:55.757050 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 06:31:55.757097 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3738,"thread_start_us":92,"threads_started":1}
I20260928 06:31:55.757532 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:55.749155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42538 (local address 127.22.225.129:34637)
0928 06:31:55.749308 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:55.749313 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:55.752891 (+ 3578us) server_negotiation.cc:407] Connection header received
0928 06:31:55.752927 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:55.752929 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:55.752943 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:55.753002 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:55.753186 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:55.754108 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:55.755890 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:55.756734 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:55.756804 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:55.756860 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:55.757418 (+ 558us) server_negotiation.cc:299] Negotiation successful
0928 06:31:55.757462 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":92,"threads_started":1}
I20260928 06:31:55.758126 7009 tablet_service.cc:1514] Processing DeleteTablet for tablet 8706ea27516c48a9a4e432a7be0742a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:31:35 UTC) from {username='kudu', principal='kudu/127.22.225.190@KRBTEST.COM'} at 127.0.0.1:42538
I20260928 06:31:55.760773 7084 tablet_replica.cc:330] stopping tablet replica
I20260928 06:31:55.760936 7084 raft_consensus.cc:2229] T 8706ea27516c48a9a4e432a7be0742a5 P 43a313fadad1418cbcebf59dd0ecb76a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:31:55.761050 7084 raft_consensus.cc:2258] T 8706ea27516c48a9a4e432a7be0742a5 P 43a313fadad1418cbcebf59dd0ecb76a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:31:55.761490 7084 ts_tablet_manager.cc:1905] T 8706ea27516c48a9a4e432a7be0742a5 P 43a313fadad1418cbcebf59dd0ecb76a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:31:55.764258 7084 ts_tablet_manager.cc:1918] T 8706ea27516c48a9a4e432a7be0742a5 P 43a313fadad1418cbcebf59dd0ecb76a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:31:55.764338 7084 log.cc:1196] T 8706ea27516c48a9a4e432a7be0742a5 P 43a313fadad1418cbcebf59dd0ecb76a: Deleting WAL directory at /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-0/wal/wals/8706ea27516c48a9a4e432a7be0742a5
I20260928 06:31:55.764616 7084 ts_tablet_manager.cc:1939] T 8706ea27516c48a9a4e432a7be0742a5 P 43a313fadad1418cbcebf59dd0ecb76a: Deleting consensus metadata
I20260928 06:31:55.765183 6874 catalog_manager.cc:4841] TS 43a313fadad1418cbcebf59dd0ecb76a (127.22.225.129:34637): tablet 8706ea27516c48a9a4e432a7be0742a5 (table test-table [id=7506e0c06c4149a7bb3731f2c443d0c0]) successfully deleted
W20260928 06:31:55.841996 7080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:55.842250 7080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:55.842293 7080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:55.843418 7080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:55.843484 7080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:55.843531 7080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.130
I20260928 06:31:55.845220 7080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:38327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab
--principal=kudu/127.22.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.130:35187
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.225.130
--webserver_port=46691
--tserver_master_addrs=127.22.225.190:39765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.130
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:31:55.845477 7080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:55.845721 7080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:55.850744 7090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:55.850872 7080 server_base.cc:1034] running on GCE node
W20260928 06:31:55.850754 7092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:55.850986 7089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:55.851199 7080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:55.851473 7080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:55.852661 7080 hybrid_clock.cc:648] HybridClock initialized: now 1790577115852640 us; error 38 us; skew 500 ppm
Sep 28 06:31:55 dist-test-slave-d59m krb5kdc[6307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577115, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:55.855396 7080 init.cc:377] Logged in from keytab as kudu/127.22.225.130@KRBTEST.COM (short username kudu)
I20260928 06:31:55.855847 7080 webserver.cc:460] Webserver started at http://127.22.225.130:46691/ using document root <none> and password file <none>
I20260928 06:31:55.856087 7080 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:55.856175 7080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:55.857695 7080 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:55.858457 7099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:55.858628 7080 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:55.858726 7080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-1/wal
uuid: "116591f9d0bd4e1590c1d52410720989"
format_stamp: "Formatted at 2026-09-28 06:31:34 on dist-test-slave-d59m"
server_key: "3cc391bbcd488f873a445eefea2ad995"
server_key_iv: "1275b3bfdfbc1ae5d3d83e3b4647f947"
server_key_version: "encryptionkey@0"
I20260928 06:31:55.859114 7080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:55.869093 7080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:55.871501 7080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:55.871668 7080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:55.871903 7080 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:31:55.872241 7080 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:31:55.872311 7080 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:55.872355 7080 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:31:55.872385 7080 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:55.879254 7080 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.130:35187
I20260928 06:31:55.879299 7212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.130:35187 every 8 connection(s)
I20260928 06:31:55.879655 7080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
I20260928 06:31:55.881608 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 7080
I20260928 06:31:55.881902 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.131:46263
--local_ip_for_outbound_sockets=127.22.225.131
--tserver_master_addrs=127.22.225.190:39765
--webserver_port=35703
--webserver_interface=127.22.225.131
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab
--principal=kudu/127.22.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:38327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 06:31:55 dist-test-slave-d59m krb5kdc[6307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577115, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.130@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:31:55.893095 7079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:55.880370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:57077 (local address 127.22.225.190:39765)
0928 06:31:55.880423 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:55.880425 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:55.880683 (+ 258us) server_negotiation.cc:407] Connection header received
0928 06:31:55.881351 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:55.881352 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:55.881368 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:55.881423 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:55.886018 (+ 4595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:55.886525 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:55.887557 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:55.887766 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:55.887875 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:55.890122 (+ 2247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:55.890124 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:55.890124 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:55.890134 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:55.891641 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:55.892063 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:55.892064 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:55.892065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:55.892124 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:55.892387 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:55.892388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:55.892388 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:55.892502 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:55.892581 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:55.892907 (+ 326us) server_negotiation.cc:299] Negotiation successful
0928 06:31:55.893004 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 06:31:55.893280 7215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:55.880369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:39765 (local address 127.22.225.130:57077)
0928 06:31:55.880522 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:55.880570 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:55.881224 (+ 654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:55.883480 (+ 2256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:55.883485 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:55.884243 (+ 758us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:55.884527 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:55.884534 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:55.886639 (+ 2105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:55.886642 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:55.887462 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:55.887464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:55.887889 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:55.887890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:55.888018 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:55.888565 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:55.888611 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:55.890019 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:55.891736 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:55.891744 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:55.891746 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:55.891977 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:55.892199 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:55.892199 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:55.892200 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:55.892313 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:55.892572 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:55.892618 (+ 46us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:55.892758 (+ 140us) client_negotiation.cc:769] Sending connection context
0928 06:31:55.892871 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 06:31:55.893032 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":75,"threads_started":1}
I20260928 06:31:55.893910 7213 heartbeater.cc:347] Connected to a master server at 127.22.225.190:39765
I20260928 06:31:55.894014 7213 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:55.894205 7213 heartbeater.cc:511] Master 127.22.225.190:39765 requested a full tablet report, sending...
I20260928 06:31:55.894575 6887 ts_manager.cc:194] Registered new tserver with Master: 116591f9d0bd4e1590c1d52410720989 (127.22.225.130:35187)
I20260928 06:31:55.894976 6887 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.225.130@KRBTEST.COM'} at 127.22.225.130:57077
W20260928 06:31:56.003875 7216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:56.004172 7216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:56.004213 7216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:56.006058 7216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:56.006129 7216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:56.006171 7216 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.131
I20260928 06:31:56.008424 7216 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:38327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab
--principal=kudu/127.22.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.131:46263
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.225.131
--webserver_port=35703
--tserver_master_addrs=127.22.225.190:39765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7216
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.131
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:31:56.008711 7216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:56.008988 7216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:56.014541 7224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:56.014533 7222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:56.014885 7216 server_base.cc:1034] running on GCE node
W20260928 06:31:56.014606 7221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:56.015184 7216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:56.015439 7216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:56.016824 7216 hybrid_clock.cc:648] HybridClock initialized: now 1790577116016784 us; error 30 us; skew 500 ppm
Sep 28 06:31:56 dist-test-slave-d59m krb5kdc[6307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577116, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:56.019667 7216 init.cc:377] Logged in from keytab as kudu/127.22.225.131@KRBTEST.COM (short username kudu)
I20260928 06:31:56.020215 7216 webserver.cc:460] Webserver started at http://127.22.225.131:35703/ using document root <none> and password file <none>
I20260928 06:31:56.020490 7216 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:56.020598 7216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:56.022408 7216 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 06:31:56.023093 7231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:56.023315 7216 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:31:56.023418 7216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-2/wal
uuid: "80789f481b4948bd9e8732283ec4db11"
format_stamp: "Formatted at 2026-09-28 06:31:35 on dist-test-slave-d59m"
server_key: "ea0c4a6e5e4502802908171df0bbb3e0"
server_key_iv: "65a6afeeee22a63c5edae87189c89117"
server_key_version: "encryptionkey@0"
I20260928 06:31:56.023880 7216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:56.037480 7216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:56.039960 7216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:56.040120 7216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:56.040385 7216 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:31:56.040774 7216 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:31:56.040825 7216 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:56.040858 7216 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:31:56.040876 7216 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:56.048745 7216 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.131:46263
I20260928 06:31:56.048834 7344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.131:46263 every 8 connection(s)
I20260928 06:31:56.049211 7216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
Sep 28 06:31:56 dist-test-slave-d59m krb5kdc[6307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577116, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.131@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:31:56.058547 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 7216
I20260928 06:31:56.059844 7079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:56.050137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:56137 (local address 127.22.225.190:39765)
0928 06:31:56.050220 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:56.050223 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:56.050456 (+ 233us) server_negotiation.cc:407] Connection header received
0928 06:31:56.051001 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:56.051002 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:56.051015 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:56.051071 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:56.052103 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:56.052677 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:56.053698 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:56.053920 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:56.053998 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:56.056476 (+ 2478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:56.056478 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:56.056478 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:56.056487 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:56.057990 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:56.058520 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:56.058522 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:56.058522 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:56.058604 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:56.058971 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:56.058973 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:56.058973 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:56.059131 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:56.059181 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:56.059630 (+ 449us) server_negotiation.cc:299] Negotiation successful
0928 06:31:56.059727 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 06:31:56.060551 7347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:56.050123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:39765 (local address 127.22.225.131:56137)
0928 06:31:56.050328 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:56.050371 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:56.050886 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:56.051142 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:56.051146 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:56.051762 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:56.051972 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:56.051977 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:56.052781 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:56.052782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:56.053598 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:56.053600 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:56.054019 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:56.054021 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:56.054150 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:56.054713 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:56.054722 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:56.056368 (+ 1646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:56.058077 (+ 1709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:56.058085 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:56.058088 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:56.058395 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:56.058730 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:56.058731 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:56.058732 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:56.058880 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:56.059240 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:56.059248 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:56.059438 (+ 190us) client_negotiation.cc:769] Sending connection context
0928 06:31:56.060105 (+ 667us) client_negotiation.cc:240] Negotiation successful
0928 06:31:56.060318 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":134,"threads_started":1}
I20260928 06:31:56.061161 7345 heartbeater.cc:347] Connected to a master server at 127.22.225.190:39765
I20260928 06:31:56.061273 7345 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:56.061489 7345 heartbeater.cc:511] Master 127.22.225.190:39765 requested a full tablet report, sending...
I20260928 06:31:56.061895 6887 ts_manager.cc:194] Registered new tserver with Master: 80789f481b4948bd9e8732283ec4db11 (127.22.225.131:46263)
I20260928 06:31:56.062348 6887 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.225.131@KRBTEST.COM'} at 127.22.225.131:56137
I20260928 06:31:56.066725 7349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:56.059112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39748 (local address 127.22.225.190:39765)
0928 06:31:56.059228 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:56.059230 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:56.059328 (+ 98us) server_negotiation.cc:407] Connection header received
0928 06:31:56.059431 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:56.059432 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:56.059445 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:56.059529 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:56.060416 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:56.061194 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:56.062343 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:56.062625 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:56.062705 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:56.064064 (+ 1359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:56.064066 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:56.064066 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:56.064073 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:56.065559 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:56.065933 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:56.065934 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:56.065934 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:56.065995 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:56.066198 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:56.066199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:56.066199 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:56.066315 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:56.066512 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:56.066549 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 06:31:56.066623 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":53,"threads_started":1}
I20260928 06:31:56.067504 23430 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:31:56.070631 7349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:56.067860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39758 (local address 127.22.225.190:39765)
0928 06:31:56.067900 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:56.067902 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:56.068023 (+ 121us) server_negotiation.cc:407] Connection header received
0928 06:31:56.068101 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:56.068102 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:56.068130 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:56.068186 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:31:56.068321 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:56.068855 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:56.070005 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:56.070183 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:56.070229 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:56.070423 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:31:56.070465 (+ 42us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0928 06:31:56.070588 (+ 123us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39758: token expired
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 06:31:56.070703 7349 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39758: token expired
W20260928 06:31:56.070873 6814 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.22.225.190:39765)
I20260928 06:31:56.077911 7349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:56.071145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39774 (local address 127.22.225.190:39765)
0928 06:31:56.071208 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:56.071211 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:56.071220 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:31:56.071244 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:56.071245 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:56.071254 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:56.071326 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:56.071834 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:56.072626 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:56.073429 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:56.073777 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:56.073846 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:56.074982 (+ 1136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:56.074983 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:56.074984 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:56.074992 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:56.076458 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:56.076885 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:56.076887 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:56.076887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:56.076953 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:56.077227 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:56.077228 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:56.077229 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:56.077360 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:56.077439 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:56.077712 (+ 273us) server_negotiation.cc:299] Negotiation successful
0928 06:31:56.077829 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 06:31:56.079113 6887 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39774:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:31:56.080758 6887 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:31:56.083135 7009 tablet_service.cc:1467] Processing CreateTablet for tablet 2f712ca4ff094c479874ee97f0d57a9d (DEFAULT_TABLE table=test-table [id=30bb88c567964bb4b812a7abcef1e8ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:31:56.083343 7009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f712ca4ff094c479874ee97f0d57a9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:56.085117 6967 tablet_bootstrap.cc:492] T 2f712ca4ff094c479874ee97f0d57a9d P 43a313fadad1418cbcebf59dd0ecb76a: Bootstrap starting.
I20260928 06:31:56.085557 6967 tablet_bootstrap.cc:654] T 2f712ca4ff094c479874ee97f0d57a9d P 43a313fadad1418cbcebf59dd0ecb76a: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:56.086243 6967 tablet_bootstrap.cc:492] T 2f712ca4ff094c479874ee97f0d57a9d P 43a313fadad1418cbcebf59dd0ecb76a: No bootstrap required, opened a new log
I20260928 06:31:56.086303 6967 ts_tablet_manager.cc:1397] T 2f712ca4ff094c479874ee97f0d57a9d P 43a313fadad1418cbcebf59dd0ecb76a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 06:31:56.086468 6967 raft_consensus.cc:348] T 2f712ca4ff094c479874ee97f0d57a9d P 43a313fadad1418cbcebf59dd0ecb76a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80789f481b4948bd9e8732283ec4db11" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 46263 } } peers { permanent_uuid: "116591f9d0bd4e1590c1d52410720989" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 35187 } } peers { permanent_uuid: "43a313fadad1418cbcebf59dd0ecb76a" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34637 } }
I20260928 06:31:56.086558 6967 raft_consensus.cc:374] T 2f712ca4ff094c479874ee97f0d57a9d P 43a313fadad1418cbcebf59dd0ecb76a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:56.086609 6967 raft_consensus.cc:729] T 2f712ca4ff094c479874ee97f0d57a9d P 43a313fadad1418cbcebf59dd0ecb76a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43a313fadad1418cbcebf59dd0ecb76a, State: Initialized, Role: FOLLOWER
I20260928 06:31:56.086665 6967 consensus_queue.cc:260] T 2f712ca4ff094c479874ee97f0d57a9d P 43a313fadad1418cbcebf59dd0ecb76a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80789f481b4948bd9e8732283ec4db11" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 46263 } } peers { permanent_uuid: "116591f9d0bd4e1590c1d52410720989" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 35187 } } peers { permanent_uuid: "43a313fadad1418cbcebf59dd0ecb76a" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34637 } }
I20260928 06:31:56.086789 6967 ts_tablet_manager.cc:1428] T 2f712ca4ff094c479874ee97f0d57a9d P 43a313fadad1418cbcebf59dd0ecb76a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:56.087360 7082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:56.082890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.131:46263 (local address 127.0.0.1:40004)
0928 06:31:56.083024 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:56.083034 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:56.083077 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:56.083595 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:56.083596 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:56.083598 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:56.083642 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:56.083643 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:56.084628 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:56.084629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:56.085684 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:56.085686 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:56.087145 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:56.087146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:56.087199 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:56.087202 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:31:56.087231 (+ 29us) client_negotiation.cc:240] Negotiation successful
0928 06:31:56.087266 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":3}
I20260928 06:31:56.087445 7354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:56.083051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58024 (local address 127.22.225.130:35187)
0928 06:31:56.083458 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:56.083462 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:56.083474 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:31:56.083514 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:56.083516 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:56.083532 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:56.083620 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:56.083718 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:56.084829 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:56.086071 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:56.086931 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:56.087004 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:56.087065 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:56.087312 (+ 247us) server_negotiation.cc:299] Negotiation successful
0928 06:31:56.087357 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":81,"threads_started":1}
I20260928 06:31:56.087607 7355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:56.082890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.130:35187 (local address 127.0.0.1:58024)
0928 06:31:56.083209 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:56.083220 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:56.083266 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:56.083626 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:56.083628 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:56.083629 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:56.083663 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:56.083665 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:56.084922 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:56.084923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:56.085979 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:56.085981 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:56.087104 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:56.087105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:56.087168 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:56.087172 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:31:56.087220 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 06:31:56.087266 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"mutex_wait_us":222,"thread_start_us":63,"threads_started":1}
I20260928 06:31:56.088160 7356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:56.083234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40004 (local address 127.22.225.131:46263)
0928 06:31:56.083421 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:56.083425 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:56.083435 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:31:56.083476 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:56.083478 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:56.083493 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:56.083712 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:56.083846 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:56.084508 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:56.085861 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:56.087029 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:56.087999 (+ 970us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:56.088045 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:56.088068 (+ 23us) server_negotiation.cc:299] Negotiation successful
0928 06:31:56.088098 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":98,"threads_started":1}
I20260928 06:31:56.088433 7147 tablet_service.cc:1467] Processing CreateTablet for tablet 2f712ca4ff094c479874ee97f0d57a9d (DEFAULT_TABLE table=test-table [id=30bb88c567964bb4b812a7abcef1e8ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:31:56.088882 7147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f712ca4ff094c479874ee97f0d57a9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:56.088953 7279 tablet_service.cc:1467] Processing CreateTablet for tablet 2f712ca4ff094c479874ee97f0d57a9d (DEFAULT_TABLE table=test-table [id=30bb88c567964bb4b812a7abcef1e8ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:31:56.089293 7279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f712ca4ff094c479874ee97f0d57a9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:56.092175 7358 tablet_bootstrap.cc:492] T 2f712ca4ff094c479874ee97f0d57a9d P 116591f9d0bd4e1590c1d52410720989: Bootstrap starting.
I20260928 06:31:56.092723 7359 tablet_bootstrap.cc:492] T 2f712ca4ff094c479874ee97f0d57a9d P 80789f481b4948bd9e8732283ec4db11: Bootstrap starting.
I20260928 06:31:56.092988 7358 tablet_bootstrap.cc:654] T 2f712ca4ff094c479874ee97f0d57a9d P 116591f9d0bd4e1590c1d52410720989: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:56.093362 7359 tablet_bootstrap.cc:654] T 2f712ca4ff094c479874ee97f0d57a9d P 80789f481b4948bd9e8732283ec4db11: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:56.093539 7358 log.cc:824] T 2f712ca4ff094c479874ee97f0d57a9d P 116591f9d0bd4e1590c1d52410720989: Log is configured to *not* fsync() on all Append() calls
I20260928 06:31:56.093894 7359 log.cc:824] T 2f712ca4ff094c479874ee97f0d57a9d P 80789f481b4948bd9e8732283ec4db11: Log is configured to *not* fsync() on all Append() calls
I20260928 06:31:56.094341 7358 tablet_bootstrap.cc:492] T 2f712ca4ff094c479874ee97f0d57a9d P 116591f9d0bd4e1590c1d52410720989: No bootstrap required, opened a new log
I20260928 06:31:56.094419 7358 ts_tablet_manager.cc:1397] T 2f712ca4ff094c479874ee97f0d57a9d P 116591f9d0bd4e1590c1d52410720989: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:31:56.094676 7359 tablet_bootstrap.cc:492] T 2f712ca4ff094c479874ee97f0d57a9d P 80789f481b4948bd9e8732283ec4db11: No bootstrap required, opened a new log
I20260928 06:31:56.094727 7359 ts_tablet_manager.cc:1397] T 2f712ca4ff094c479874ee97f0d57a9d P 80789f481b4948bd9e8732283ec4db11: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:31:56.096243 7358 raft_consensus.cc:348] T 2f712ca4ff094c479874ee97f0d57a9d P 116591f9d0bd4e1590c1d52410720989 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80789f481b4948bd9e8732283ec4db11" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 46263 } } peers { permanent_uuid: "116591f9d0bd4e1590c1d52410720989" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 35187 } } peers { permanent_uuid: "43a313fadad1418cbcebf59dd0ecb76a" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34637 } }
I20260928 06:31:56.096410 7358 raft_consensus.cc:374] T 2f712ca4ff094c479874ee97f0d57a9d P 116591f9d0bd4e1590c1d52410720989 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:56.096333 7359 raft_consensus.cc:348] T 2f712ca4ff094c479874ee97f0d57a9d P 80789f481b4948bd9e8732283ec4db11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80789f481b4948bd9e8732283ec4db11" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 46263 } } peers { permanent_uuid: "116591f9d0bd4e1590c1d52410720989" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 35187 } } peers { permanent_uuid: "43a313fadad1418cbcebf59dd0ecb76a" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34637 } }
I20260928 06:31:56.096458 7358 raft_consensus.cc:729] T 2f712ca4ff094c479874ee97f0d57a9d P 116591f9d0bd4e1590c1d52410720989 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 116591f9d0bd4e1590c1d52410720989, State: Initialized, Role: FOLLOWER
I20260928 06:31:56.096499 7359 raft_consensus.cc:374] T 2f712ca4ff094c479874ee97f0d57a9d P 80789f481b4948bd9e8732283ec4db11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:56.096544 7359 raft_consensus.cc:729] T 2f712ca4ff094c479874ee97f0d57a9d P 80789f481b4948bd9e8732283ec4db11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80789f481b4948bd9e8732283ec4db11, State: Initialized, Role: FOLLOWER
I20260928 06:31:56.096637 7358 consensus_queue.cc:260] T 2f712ca4ff094c479874ee97f0d57a9d P 116591f9d0bd4e1590c1d52410720989 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80789f481b4948bd9e8732283ec4db11" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 46263 } } peers { permanent_uuid: "116591f9d0bd4e1590c1d52410720989" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 35187 } } peers { permanent_uuid: "43a313fadad1418cbcebf59dd0ecb76a" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34637 } }
I20260928 06:31:56.096679 7359 consensus_queue.cc:260] T 2f712ca4ff094c479874ee97f0d57a9d P 80789f481b4948bd9e8732283ec4db11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80789f481b4948bd9e8732283ec4db11" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 46263 } } peers { permanent_uuid: "116591f9d0bd4e1590c1d52410720989" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 35187 } } peers { permanent_uuid: "43a313fadad1418cbcebf59dd0ecb76a" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34637 } }
I20260928 06:31:56.096931 7358 ts_tablet_manager.cc:1428] T 2f712ca4ff094c479874ee97f0d57a9d P 116591f9d0bd4e1590c1d52410720989: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260928 06:31:56.096971 7359 ts_tablet_manager.cc:1428] T 2f712ca4ff094c479874ee97f0d57a9d P 80789f481b4948bd9e8732283ec4db11: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 06:31:56.096982 7213 heartbeater.cc:503] Master 127.22.225.190:39765 was elected leader, sending a full tablet report...
I20260928 06:31:56.097121 7345 heartbeater.cc:503] Master 127.22.225.190:39765 was elected leader, sending a full tablet report...
I20260928 06:31:56.100962 7363 raft_consensus.cc:482] T 2f712ca4ff094c479874ee97f0d57a9d P 80789f481b4948bd9e8732283ec4db11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:31:56.101030 7363 raft_consensus.cc:504] T 2f712ca4ff094c479874ee97f0d57a9d P 80789f481b4948bd9e8732283ec4db11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80789f481b4948bd9e8732283ec4db11" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 46263 } } peers { permanent_uuid: "116591f9d0bd4e1590c1d52410720989" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 35187 } } peers { permanent_uuid: "43a313fadad1418cbcebf59dd0ecb76a" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34637 } }
I20260928 06:31:56.101576 7363 leader_election.cc:288] T 2f712ca4ff094c479874ee97f0d57a9d P 80789f481b4948bd9e8732283ec4db11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 116591f9d0bd4e1590c1d52410720989 (127.22.225.130:35187), 43a313fadad1418cbcebf59dd0ecb76a (127.22.225.129:34637)
I20260928 06:31:56.104991 7347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:56.101521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.129:34637 (local address 127.22.225.131:40569)
0928 06:31:56.101572 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:56.101587 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:56.101638 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:56.101887 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:56.101889 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:56.101891 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:56.101961 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:56.101963 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:56.102904 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:56.102906 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:56.104113 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:56.104116 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:56.104732 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:56.104733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:56.104826 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:56.104830 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:31:56.104877 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 06:31:56.104918 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":31}
I20260928 06:31:56.105265 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:56.101673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:40569 (local address 127.22.225.129:34637)
0928 06:31:56.101722 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:56.101724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:56.101734 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:31:56.101763 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:56.101764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:56.101774 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:56.101834 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:56.102039 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:56.102787 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:56.104204 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:56.104641 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:56.104724 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:56.104735 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:56.105147 (+ 412us) server_negotiation.cc:299] Negotiation successful
0928 06:31:56.105197 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 06:31:56.105751 7029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f712ca4ff094c479874ee97f0d57a9d" candidate_uuid: "80789f481b4948bd9e8732283ec4db11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43a313fadad1418cbcebf59dd0ecb76a" is_pre_election: true
I20260928 06:31:56.105832 7354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:56.101763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:57657 (local address 127.22.225.130:35187)
0928 06:31:56.101810 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:56.101812 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:56.101898 (+ 86us) server_negotiation.cc:407] Connection header received
0928 06:31:56.101950 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:56.101950 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:56.101960 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:56.102010 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:56.102204 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:56.102843 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:56.104854 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:56.105424 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:56.105475 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:56.105484 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:56.105716 (+ 232us) server_negotiation.cc:299] Negotiation successful
0928 06:31:56.105761 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 06:31:56.105940 7029 raft_consensus.cc:2454] T 2f712ca4ff094c479874ee97f0d57a9d P 43a313fadad1418cbcebf59dd0ecb76a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80789f481b4948bd9e8732283ec4db11 in term 0.
I20260928 06:31:56.106087 7364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:56.101697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.130:35187 (local address 127.22.225.131:57657)
0928 06:31:56.101832 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:56.101854 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:56.101908 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:56.102055 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:56.102056 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:56.102057 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:56.102113 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:56.102114 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:56.102941 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:56.102941 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:56.104755 (+ 1814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:56.104757 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:56.105546 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:56.105547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:56.105627 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:56.105630 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:31:56.105940 (+ 310us) client_negotiation.cc:240] Negotiation successful
0928 06:31:56.105990 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":1,"thread_start_us":85,"threads_started":1}
I20260928 06:31:56.106168 7235 leader_election.cc:302] T 2f712ca4ff094c479874ee97f0d57a9d P 80789f481b4948bd9e8732283ec4db11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43a313fadad1418cbcebf59dd0ecb76a, 80789f481b4948bd9e8732283ec4db11; no voters:
I20260928 06:31:56.106297 7363 raft_consensus.cc:2781] T 2f712ca4ff094c479874ee97f0d57a9d P 80789f481b4948bd9e8732283ec4db11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:31:56.106348 7363 raft_consensus.cc:482] T 2f712ca4ff094c479874ee97f0d57a9d P 80789f481b4948bd9e8732283ec4db11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:31:56.106371 7363 raft_consensus.cc:3037] T 2f712ca4ff094c479874ee97f0d57a9d P 80789f481b4948bd9e8732283ec4db11 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:56.106521 7167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f712ca4ff094c479874ee97f0d57a9d" candidate_uuid: "80789f481b4948bd9e8732283ec4db11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "116591f9d0bd4e1590c1d52410720989" is_pre_election: true
I20260928 06:31:56.106658 7167 raft_consensus.cc:2454] T 2f712ca4ff094c479874ee97f0d57a9d P 116591f9d0bd4e1590c1d52410720989 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80789f481b4948bd9e8732283ec4db11 in term 0.
I20260928 06:31:56.107218 7363 raft_consensus.cc:504] T 2f712ca4ff094c479874ee97f0d57a9d P 80789f481b4948bd9e8732283ec4db11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80789f481b4948bd9e8732283ec4db11" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 46263 } } peers { permanent_uuid: "116591f9d0bd4e1590c1d52410720989" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 35187 } } peers { permanent_uuid: "43a313fadad1418cbcebf59dd0ecb76a" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34637 } }
I20260928 06:31:56.107429 7363 leader_election.cc:288] T 2f712ca4ff094c479874ee97f0d57a9d P 80789f481b4948bd9e8732283ec4db11 [CANDIDATE]: Term 1 election: Requested vote from peers 116591f9d0bd4e1590c1d52410720989 (127.22.225.130:35187), 43a313fadad1418cbcebf59dd0ecb76a (127.22.225.129:34637)
I20260928 06:31:56.107478 7167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f712ca4ff094c479874ee97f0d57a9d" candidate_uuid: "80789f481b4948bd9e8732283ec4db11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "116591f9d0bd4e1590c1d52410720989"
I20260928 06:31:56.107553 7167 raft_consensus.cc:3037] T 2f712ca4ff094c479874ee97f0d57a9d P 116591f9d0bd4e1590c1d52410720989 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:56.107560 7029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f712ca4ff094c479874ee97f0d57a9d" candidate_uuid: "80789f481b4948bd9e8732283ec4db11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43a313fadad1418cbcebf59dd0ecb76a"
I20260928 06:31:56.107635 7029 raft_consensus.cc:3037] T 2f712ca4ff094c479874ee97f0d57a9d P 43a313fadad1418cbcebf59dd0ecb76a [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:56.108381 7029 raft_consensus.cc:2454] T 2f712ca4ff094c479874ee97f0d57a9d P 43a313fadad1418cbcebf59dd0ecb76a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80789f481b4948bd9e8732283ec4db11 in term 1.
I20260928 06:31:56.108464 7167 raft_consensus.cc:2454] T 2f712ca4ff094c479874ee97f0d57a9d P 116591f9d0bd4e1590c1d52410720989 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80789f481b4948bd9e8732283ec4db11 in term 1.
I20260928 06:31:56.108531 7235 leader_election.cc:302] T 2f712ca4ff094c479874ee97f0d57a9d P 80789f481b4948bd9e8732283ec4db11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43a313fadad1418cbcebf59dd0ecb76a, 80789f481b4948bd9e8732283ec4db11; no voters:
I20260928 06:31:56.108688 7363 raft_consensus.cc:2781] T 2f712ca4ff094c479874ee97f0d57a9d P 80789f481b4948bd9e8732283ec4db11 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:31:56.108870 7363 raft_consensus.cc:686] T 2f712ca4ff094c479874ee97f0d57a9d P 80789f481b4948bd9e8732283ec4db11 [term 1 LEADER]: Becoming Leader. State: Replica: 80789f481b4948bd9e8732283ec4db11, State: Running, Role: LEADER
I20260928 06:31:56.108971 7363 consensus_queue.cc:237] T 2f712ca4ff094c479874ee97f0d57a9d P 80789f481b4948bd9e8732283ec4db11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80789f481b4948bd9e8732283ec4db11" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 46263 } } peers { permanent_uuid: "116591f9d0bd4e1590c1d52410720989" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 35187 } } peers { permanent_uuid: "43a313fadad1418cbcebf59dd0ecb76a" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34637 } }
I20260928 06:31:56.109777 6887 catalog_manager.cc:5495] T 2f712ca4ff094c479874ee97f0d57a9d P 80789f481b4948bd9e8732283ec4db11 reported cstate change: term changed from 0 to 1, leader changed from <none> to 80789f481b4948bd9e8732283ec4db11 (127.22.225.131). New cstate: current_term: 1 leader_uuid: "80789f481b4948bd9e8732283ec4db11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80789f481b4948bd9e8732283ec4db11" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 46263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "116591f9d0bd4e1590c1d52410720989" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 35187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43a313fadad1418cbcebf59dd0ecb76a" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34637 } health_report { overall_health: UNKNOWN } } }
I20260928 06:31:56.115545 7356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:56.113015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40016 (local address 127.22.225.131:46263)
0928 06:31:56.113055 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:56.113056 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:56.113067 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:31:56.113152 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:56.113153 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:56.113166 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:56.113223 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:31:56.113361 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:56.113804 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:56.114837 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:56.115012 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:56.115066 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:56.115226 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:31:56.115314 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:31:56.115372 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:56.115433 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 06:31:56.115448 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:31:56.117376 7029 raft_consensus.cc:1261] T 2f712ca4ff094c479874ee97f0d57a9d P 43a313fadad1418cbcebf59dd0ecb76a [term 1 FOLLOWER]: Refusing update from remote peer 80789f481b4948bd9e8732283ec4db11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:31:56.117376 7167 raft_consensus.cc:1261] T 2f712ca4ff094c479874ee97f0d57a9d P 116591f9d0bd4e1590c1d52410720989 [term 1 FOLLOWER]: Refusing update from remote peer 80789f481b4948bd9e8732283ec4db11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:31:56.117699 7363 consensus_queue.cc:1035] T 2f712ca4ff094c479874ee97f0d57a9d P 80789f481b4948bd9e8732283ec4db11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "116591f9d0bd4e1590c1d52410720989" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 35187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:31:56.117844 7365 consensus_queue.cc:1035] T 2f712ca4ff094c479874ee97f0d57a9d P 80789f481b4948bd9e8732283ec4db11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43a313fadad1418cbcebf59dd0ecb76a" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:31:56.120162 7367 mvcc.cc:205] Tried to move back new op lower bound from 7334203867614281728 to 7334203867583098880. Current Snapshot: MvccSnapshot[applied={T|T < 7334203867614281728}]
I20260928 06:31:56.120649 7368 mvcc.cc:205] Tried to move back new op lower bound from 7334203867614281728 to 7334203867583098880. Current Snapshot: MvccSnapshot[applied={T|T < 7334203867614281728}]
I20260928 06:31:56.120748 7369 mvcc.cc:205] Tried to move back new op lower bound from 7334203867614281728 to 7334203867583098880. Current Snapshot: MvccSnapshot[applied={T|T < 7334203867614281728}]
I20260928 06:31:56.122452 6886 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:31:56.122567 6886 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:31:56.123238 6886 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c3eacb4d322f415eb7bb4b021abfbece: Sending DeleteTablet for 3 replicas of tablet 2f712ca4ff094c479874ee97f0d57a9d
I20260928 06:31:56.123495 7009 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f712ca4ff094c479874ee97f0d57a9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:31:56 UTC) from {username='kudu', principal='kudu/127.22.225.190@KRBTEST.COM'} at 127.0.0.1:42538
I20260928 06:31:56.123593 7084 tablet_replica.cc:330] T 2f712ca4ff094c479874ee97f0d57a9d P 43a313fadad1418cbcebf59dd0ecb76a: stopping tablet replica
I20260928 06:31:56.123651 7084 raft_consensus.cc:2229] T 2f712ca4ff094c479874ee97f0d57a9d P 43a313fadad1418cbcebf59dd0ecb76a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:31:56.123641 7147 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f712ca4ff094c479874ee97f0d57a9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:31:56 UTC) from {username='kudu', principal='kudu/127.22.225.190@KRBTEST.COM'} at 127.0.0.1:58024
I20260928 06:31:56.123700 7084 raft_consensus.cc:2258] T 2f712ca4ff094c479874ee97f0d57a9d P 43a313fadad1418cbcebf59dd0ecb76a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:31:56.123818 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 6925
I20260928 06:31:56.123855 7279 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f712ca4ff094c479874ee97f0d57a9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:31:56 UTC) from {username='kudu', principal='kudu/127.22.225.190@KRBTEST.COM'} at 127.0.0.1:40004
I20260928 06:31:56.128336 7378 tablet_replica.cc:330] T 2f712ca4ff094c479874ee97f0d57a9d P 80789f481b4948bd9e8732283ec4db11: stopping tablet replica
I20260928 06:31:56.128481 7378 raft_consensus.cc:2229] T 2f712ca4ff094c479874ee97f0d57a9d P 80789f481b4948bd9e8732283ec4db11 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:31:56.128610 7378 raft_consensus.cc:2258] T 2f712ca4ff094c479874ee97f0d57a9d P 80789f481b4948bd9e8732283ec4db11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:31:56.128731 7377 tablet_replica.cc:330] T 2f712ca4ff094c479874ee97f0d57a9d P 116591f9d0bd4e1590c1d52410720989: stopping tablet replica
I20260928 06:31:56.128887 7377 raft_consensus.cc:2229] T 2f712ca4ff094c479874ee97f0d57a9d P 116591f9d0bd4e1590c1d52410720989 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:31:56.129038 7377 raft_consensus.cc:2258] T 2f712ca4ff094c479874ee97f0d57a9d P 116591f9d0bd4e1590c1d52410720989 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:31:56.129925 7377 ts_tablet_manager.cc:1905] T 2f712ca4ff094c479874ee97f0d57a9d P 116591f9d0bd4e1590c1d52410720989: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 06:31:56.130153 6874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.225.129:34637 (error 108)
I20260928 06:31:56.130316 7378 ts_tablet_manager.cc:1905] T 2f712ca4ff094c479874ee97f0d57a9d P 80789f481b4948bd9e8732283ec4db11: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 06:31:56.130947 7355 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:31:56.130689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.129:34637 (local address 127.0.0.1:42554)
0928 06:31:56.130739 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:56.130750 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:56.130789 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:56.130898 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.129:34637: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 06:31:56.131075 6874 catalog_manager.cc:4584] TS 43a313fadad1418cbcebf59dd0ecb76a (127.22.225.129:34637): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2f712ca4ff094c479874ee97f0d57a9d: Network error: Client connection negotiation failed: client connection to 127.22.225.129:34637: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 06:31:56.131995 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 7080
I20260928 06:31:56.137847 7378 ts_tablet_manager.cc:1918] T 2f712ca4ff094c479874ee97f0d57a9d P 80789f481b4948bd9e8732283ec4db11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:31:56.138001 7378 log.cc:1196] T 2f712ca4ff094c479874ee97f0d57a9d P 80789f481b4948bd9e8732283ec4db11: Deleting WAL directory at /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-2/wal/wals/2f712ca4ff094c479874ee97f0d57a9d
I20260928 06:31:56.138227 7355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:56.138093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.130:35187 (local address 127.0.0.1:58032)
0928 06:31:56.138140 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:56.138192 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.130:35187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 06:31:56.138521 7378 ts_tablet_manager.cc:1939] T 2f712ca4ff094c479874ee97f0d57a9d P 80789f481b4948bd9e8732283ec4db11: Deleting consensus metadata
I20260928 06:31:56.138516 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 7216
I20260928 06:31:56.143934 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 6854
I20260928 06:31:56.144006 7355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:56.143878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.131:46263 (local address 127.0.0.1:40032)
0928 06:31:56.143922 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:56.143948 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.131:46263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260928 06:32:16.151928 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:39765
--webserver_interface=127.22.225.190
--webserver_port=45977
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab
--principal=kudu/127.22.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:38327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.225.190:39765
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:32:16.236749 7380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:32:16.236967 7380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:32:16.236991 7380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:32:16.238193 7380 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 06:32:16.238235 7380 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 06:32:16.238250 7380 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 06:32:16.238260 7380 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:32:16.238274 7380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:32:16.238315 7380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:32:16.238346 7380 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:32:16.238363 7380 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:32:16.239878 7380 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:38327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.22.225.190:39765
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab
--principal=kudu/127.22.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:39765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.225.190
--webserver_port=45977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7380
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:32:16.240163 7380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:32:16.240411 7380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:32:16.245647 7386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:16.245656 7388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:32:16.245828 7380 server_base.cc:1034] running on GCE node
W20260928 06:32:16.245959 7385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:32:16.246254 7380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:32:16.246618 7380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:32:16.247807 7380 hybrid_clock.cc:648] HybridClock initialized: now 1790577136247776 us; error 45 us; skew 500 ppm
Sep 28 06:32:16 dist-test-slave-d59m krb5kdc[6307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577136, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:32:16.262215 7380 init.cc:377] Logged in from keytab as kudu/127.22.225.190@KRBTEST.COM (short username kudu)
I20260928 06:32:16.262784 7380 webserver.cc:460] Webserver started at http://127.22.225.190:45977/ using document root <none> and password file <none>
I20260928 06:32:16.263104 7380 fs_manager.cc:359] Metadata directory not provided
I20260928 06:32:16.263185 7380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:32:16.264719 7380 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 06:32:16.265515 7395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:16.265745 7380 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:32:16.265838 7380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/master-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/master-0/wal
uuid: "c3eacb4d322f415eb7bb4b021abfbece"
format_stamp: "Formatted at 2026-09-28 06:31:34 on dist-test-slave-d59m"
server_key: "0ae844b205141afacca93022773b09dd"
server_key_iv: "88b2d78127515b9607f76597944b916c"
server_key_version: "encryptionkey@0"
I20260928 06:32:16.266294 7380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:32:16.310734 7380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:32:16.313220 7380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:32:16.313367 7380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:32:16.317572 7380 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.190:39765
I20260928 06:32:16.317678 7447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.190:39765 every 8 connection(s)
I20260928 06:32:16.318046 7380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
I20260928 06:32:16.319996 7448 sys_catalog.cc:263] Verifying existing consensus state
I20260928 06:32:16.320885 7448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3eacb4d322f415eb7bb4b021abfbece: Bootstrap starting.
I20260928 06:32:16.328063 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 7380
I20260928 06:32:16.328351 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.129:34637
--local_ip_for_outbound_sockets=127.22.225.129
--tserver_master_addrs=127.22.225.190:39765
--webserver_port=43921
--webserver_interface=127.22.225.129
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab
--principal=kudu/127.22.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:38327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:32:16.332089 7448 log.cc:824] T 00000000000000000000000000000000 P c3eacb4d322f415eb7bb4b021abfbece: Log is configured to *not* fsync() on all Append() calls
I20260928 06:32:16.335264 7448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3eacb4d322f415eb7bb4b021abfbece: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:32:16.335677 7448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3eacb4d322f415eb7bb4b021abfbece: Bootstrap complete.
I20260928 06:32:16.338901 7448 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3eacb4d322f415eb7bb4b021abfbece [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3eacb4d322f415eb7bb4b021abfbece" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 39765 } }
I20260928 06:32:16.339259 7448 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3eacb4d322f415eb7bb4b021abfbece [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3eacb4d322f415eb7bb4b021abfbece, State: Initialized, Role: FOLLOWER
I20260928 06:32:16.339470 7448 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3eacb4d322f415eb7bb4b021abfbece [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3eacb4d322f415eb7bb4b021abfbece" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 39765 } }
I20260928 06:32:16.339577 7448 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3eacb4d322f415eb7bb4b021abfbece [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:32:16.339655 7448 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3eacb4d322f415eb7bb4b021abfbece [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:32:16.339736 7448 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3eacb4d322f415eb7bb4b021abfbece [term 2 FOLLOWER]: Advancing to term 3
I20260928 06:32:16.341172 7448 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3eacb4d322f415eb7bb4b021abfbece [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3eacb4d322f415eb7bb4b021abfbece" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 39765 } }
I20260928 06:32:16.341305 7448 leader_election.cc:302] T 00000000000000000000000000000000 P c3eacb4d322f415eb7bb4b021abfbece [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3eacb4d322f415eb7bb4b021abfbece; no voters:
I20260928 06:32:16.341523 7448 leader_election.cc:288] T 00000000000000000000000000000000 P c3eacb4d322f415eb7bb4b021abfbece [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 06:32:16.341583 7452 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3eacb4d322f415eb7bb4b021abfbece [term 3 FOLLOWER]: Leader election won for term 3
I20260928 06:32:16.341892 7452 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3eacb4d322f415eb7bb4b021abfbece [term 3 LEADER]: Becoming Leader. State: Replica: c3eacb4d322f415eb7bb4b021abfbece, State: Running, Role: LEADER
I20260928 06:32:16.341957 7448 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3eacb4d322f415eb7bb4b021abfbece [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:32:16.342036 7452 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3eacb4d322f415eb7bb4b021abfbece [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3eacb4d322f415eb7bb4b021abfbece" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 39765 } }
I20260928 06:32:16.342386 7452 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3eacb4d322f415eb7bb4b021abfbece [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "c3eacb4d322f415eb7bb4b021abfbece" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3eacb4d322f415eb7bb4b021abfbece" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 39765 } } }
I20260928 06:32:16.342512 7452 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3eacb4d322f415eb7bb4b021abfbece [sys.catalog]: This master's current role is: LEADER
I20260928 06:32:16.342677 7452 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3eacb4d322f415eb7bb4b021abfbece [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3eacb4d322f415eb7bb4b021abfbece. Latest consensus state: current_term: 3 leader_uuid: "c3eacb4d322f415eb7bb4b021abfbece" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3eacb4d322f415eb7bb4b021abfbece" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 39765 } } }
I20260928 06:32:16.342769 7452 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3eacb4d322f415eb7bb4b021abfbece [sys.catalog]: This master's current role is: LEADER
I20260928 06:32:16.342907 7461 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:32:16.343791 7461 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:32:16.344173 7461 catalog_manager.cc:1259] Loaded cluster ID: ec1aca84d3834c95a23cf3dea47cef2e
I20260928 06:32:16.344252 7461 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:32:16.345553 7461 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:32:16.345777 7461 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c3eacb4d322f415eb7bb4b021abfbece: Loaded TSK: 0
I20260928 06:32:16.345865 7461 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c3eacb4d322f415eb7bb4b021abfbece: Loaded TSK: 1
F20260928 06:32:16.346148 7461 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fcc184539ad google::LogMessage::Fail() at ??:0
@ 0x7fcc1845688a google::LogMessage::SendToLog() at ??:0
@ 0x7fcc184534de google::LogMessage::Flush() at ??:0
@ 0x7fcc18454a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fcc1abc0104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fcc1abbdd41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fcc203ad1cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fcc203ad48e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fcc203ad594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fcc20373c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fcc2037dad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fcc1a3210d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fcc1a316954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fcc1b17a6db start_thread at ??:0
@ 0x7fcc16f5161f clone at ??:0
W20260928 06:32:16.452380 7450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:32:16.452679 7450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:32:16.452718 7450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:32:16.454238 7450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:32:16.454293 7450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:32:16.454329 7450 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.129
I20260928 06:32:16.456219 7450 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:38327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab
--principal=kudu/127.22.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.129:34637
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.225.129
--webserver_port=43921
--tserver_master_addrs=127.22.225.190:39765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.129
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:32:16.456465 7450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:32:16.456748 7450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:32:16.462087 7489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:16.462100 7490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:16.462100 7492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:32:16.462962 7450 server_base.cc:1034] running on GCE node
I20260928 06:32:16.463165 7450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:32:16.463402 7450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:32:16.464560 7450 hybrid_clock.cc:648] HybridClock initialized: now 1790577136464541 us; error 34 us; skew 500 ppm
Sep 28 06:32:16 dist-test-slave-d59m krb5kdc[6307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577136, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:32:16.467649 7450 init.cc:377] Logged in from keytab as kudu/127.22.225.129@KRBTEST.COM (short username kudu)
I20260928 06:32:16.468158 7450 webserver.cc:460] Webserver started at http://127.22.225.129:43921/ using document root <none> and password file <none>
I20260928 06:32:16.468390 7450 fs_manager.cc:359] Metadata directory not provided
I20260928 06:32:16.468446 7450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:32:16.470119 7450 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:32:16.470985 7499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:16.471169 7450 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:32:16.471277 7450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-0/wal
uuid: "43a313fadad1418cbcebf59dd0ecb76a"
format_stamp: "Formatted at 2026-09-28 06:31:34 on dist-test-slave-d59m"
server_key: "ed44e41bb9338f5afaa97a8b586735cd"
server_key_iv: "ae625cd819ee12cd936dcbf37b7c62df"
server_key_version: "encryptionkey@0"
I20260928 06:32:16.471719 7450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:32:16.491649 7450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:32:16.494521 7450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:32:16.494716 7450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:32:16.494987 7450 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:32:16.495525 7506 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 06:32:16.497550 7450 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 06:32:16.497607 7450 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 06:32:16.497639 7450 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 06:32:16.498420 7450 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 06:32:16.498468 7450 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 06:32:16.498647 7506 tablet_bootstrap.cc:492] T 2f712ca4ff094c479874ee97f0d57a9d P 43a313fadad1418cbcebf59dd0ecb76a: Bootstrap starting.
I20260928 06:32:16.506035 7450 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.129:34637
I20260928 06:32:16.506108 7613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.129:34637 every 8 connection(s)
I20260928 06:32:16.506476 7450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
I20260928 06:32:16.508185 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 7450
I20260928 06:32:16.508492 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.130:35187
--local_ip_for_outbound_sockets=127.22.225.130
--tserver_master_addrs=127.22.225.190:39765
--webserver_port=46691
--webserver_interface=127.22.225.130
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab
--principal=kudu/127.22.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:38327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:32:16.512924 7506 log.cc:824] T 2f712ca4ff094c479874ee97f0d57a9d P 43a313fadad1418cbcebf59dd0ecb76a: Log is configured to *not* fsync() on all Append() calls
I20260928 06:32:16.516156 7506 tablet_bootstrap.cc:492] T 2f712ca4ff094c479874ee97f0d57a9d P 43a313fadad1418cbcebf59dd0ecb76a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:32:16.516480 7506 tablet_bootstrap.cc:492] T 2f712ca4ff094c479874ee97f0d57a9d P 43a313fadad1418cbcebf59dd0ecb76a: Bootstrap complete.
I20260928 06:32:16.517879 7506 ts_tablet_manager.cc:1397] T 2f712ca4ff094c479874ee97f0d57a9d P 43a313fadad1418cbcebf59dd0ecb76a: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.011s
I20260928 06:32:16.519826 7506 raft_consensus.cc:348] T 2f712ca4ff094c479874ee97f0d57a9d P 43a313fadad1418cbcebf59dd0ecb76a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80789f481b4948bd9e8732283ec4db11" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 46263 } } peers { permanent_uuid: "116591f9d0bd4e1590c1d52410720989" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 35187 } } peers { permanent_uuid: "43a313fadad1418cbcebf59dd0ecb76a" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34637 } }
I20260928 06:32:16.520264 7506 raft_consensus.cc:729] T 2f712ca4ff094c479874ee97f0d57a9d P 43a313fadad1418cbcebf59dd0ecb76a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43a313fadad1418cbcebf59dd0ecb76a, State: Initialized, Role: FOLLOWER
I20260928 06:32:16.520465 7506 consensus_queue.cc:260] T 2f712ca4ff094c479874ee97f0d57a9d P 43a313fadad1418cbcebf59dd0ecb76a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80789f481b4948bd9e8732283ec4db11" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 46263 } } peers { permanent_uuid: "116591f9d0bd4e1590c1d52410720989" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 35187 } } peers { permanent_uuid: "43a313fadad1418cbcebf59dd0ecb76a" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34637 } }
I20260928 06:32:16.520826 7506 ts_tablet_manager.cc:1428] T 2f712ca4ff094c479874ee97f0d57a9d P 43a313fadad1418cbcebf59dd0ecb76a: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
W20260928 06:32:16.579952 7616 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:32:16.507196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:39765 (local address 127.22.225.129:40245)
0928 06:32:16.507413 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:16.507463 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 06:32:16.508037 (+ 574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:32:16.579732 (+ 71695us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:39765: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":102,"threads_started":1}
W20260928 06:32:16.580302 7614 heartbeater.cc:644] Failed to heartbeat to 127.22.225.190:39765 (0 consecutive failures): Network error: Failed to ping master at 127.22.225.190:39765: Client connection negotiation failed: client connection to 127.22.225.190:39765: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 06:32:16.580701 7616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:16.580550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:39765 (local address 127.22.225.129:39989)
0928 06:32:16.580627 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:16.580679 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:39765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 06:32:16.581099 7616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:16.581003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:39765 (local address 127.22.225.129:47173)
0928 06:32:16.581053 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:16.581078 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:39765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 06:32:16.581256 7614 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 06:32:16.601547 7617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:32:16.601774 7617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:32:16.601799 7617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:32:16.602993 7617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:32:16.603041 7617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:32:16.603075 7617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.130
I20260928 06:32:16.604569 7617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:38327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab
--principal=kudu/127.22.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.130:35187
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.225.130
--webserver_port=46691
--tserver_master_addrs=127.22.225.190:39765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.130
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:32:16.604879 7617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:32:16.605131 7617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:32:16.610110 7624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:16.610128 7625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:16.610208 7627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:32:16.610860 7617 server_base.cc:1034] running on GCE node
I20260928 06:32:16.611092 7617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:32:16.611368 7617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:32:16.612555 7617 hybrid_clock.cc:648] HybridClock initialized: now 1790577136612541 us; error 28 us; skew 500 ppm
Sep 28 06:32:16 dist-test-slave-d59m krb5kdc[6307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577136, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:32:16.615537 7617 init.cc:377] Logged in from keytab as kudu/127.22.225.130@KRBTEST.COM (short username kudu)
I20260928 06:32:16.615988 7617 webserver.cc:460] Webserver started at http://127.22.225.130:46691/ using document root <none> and password file <none>
I20260928 06:32:16.616243 7617 fs_manager.cc:359] Metadata directory not provided
I20260928 06:32:16.616314 7617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:32:16.617830 7617 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:32:16.618705 7634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:16.618901 7617 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:32:16.619005 7617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-1/wal
uuid: "116591f9d0bd4e1590c1d52410720989"
format_stamp: "Formatted at 2026-09-28 06:31:34 on dist-test-slave-d59m"
server_key: "3cc391bbcd488f873a445eefea2ad995"
server_key_iv: "1275b3bfdfbc1ae5d3d83e3b4647f947"
server_key_version: "encryptionkey@0"
I20260928 06:32:16.619405 7617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:32:16.656635 7617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:32:16.659138 7617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:32:16.659309 7617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:32:16.659564 7617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:32:16.660029 7641 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260928 06:32:16.662012 7641 ts_tablet_manager.cc:1852] T 2f712ca4ff094c479874ee97f0d57a9d P 116591f9d0bd4e1590c1d52410720989: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260928 06:32:16.662395 7641 ts_tablet_manager.cc:1905] T 2f712ca4ff094c479874ee97f0d57a9d P 116591f9d0bd4e1590c1d52410720989: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:32:16.665195 7641 ts_tablet_manager.cc:1918] T 2f712ca4ff094c479874ee97f0d57a9d P 116591f9d0bd4e1590c1d52410720989: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:32:16.665287 7641 log.cc:1196] T 2f712ca4ff094c479874ee97f0d57a9d P 116591f9d0bd4e1590c1d52410720989: Deleting WAL directory at /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-1/wal/wals/2f712ca4ff094c479874ee97f0d57a9d
I20260928 06:32:16.665587 7641 ts_tablet_manager.cc:1939] T 2f712ca4ff094c479874ee97f0d57a9d P 116591f9d0bd4e1590c1d52410720989: Deleting consensus metadata
I20260928 06:32:16.665804 7617 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260928 06:32:16.665867 7617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260928 06:32:16.665915 7617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:32:16.665944 7617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:16.673172 7617 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.130:35187
I20260928 06:32:16.673264 7748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.130:35187 every 8 connection(s)
I20260928 06:32:16.673565 7617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
I20260928 06:32:16.674651 7751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:16.674254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:39765 (local address 127.22.225.130:50967)
0928 06:32:16.674423 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:16.674504 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:39765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":93,"threads_started":1}
W20260928 06:32:16.674925 7749 heartbeater.cc:644] Failed to heartbeat to 127.22.225.190:39765 (0 consecutive failures): Network error: Failed to ping master at 127.22.225.190:39765: Client connection negotiation failed: client connection to 127.22.225.190:39765: connect: Connection refused (error 111)
I20260928 06:32:16.675010 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 7617
I20260928 06:32:16.675217 7751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:16.675122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:39765 (local address 127.22.225.130:46095)
0928 06:32:16.675168 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:16.675193 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:39765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 06:32:16.675297 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.131:46263
--local_ip_for_outbound_sockets=127.22.225.131
--tserver_master_addrs=127.22.225.190:39765
--webserver_port=35703
--webserver_interface=127.22.225.131
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab
--principal=kudu/127.22.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:38327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:32:16.675526 7751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:16.675446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:39765 (local address 127.22.225.130:33871)
0928 06:32:16.675481 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:16.675501 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:39765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260928 06:32:16.675612 7749 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 06:32:16.760005 7752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:32:16.760241 7752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:32:16.760277 7752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:32:16.761544 7752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:32:16.761615 7752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:32:16.761660 7752 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.131
I20260928 06:32:16.763185 7752 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:38327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab
--principal=kudu/127.22.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.131:46263
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.225.131
--webserver_port=35703
--tserver_master_addrs=127.22.225.190:39765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7752
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.131
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:32:16.763463 7752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:32:16.763703 7752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:32:16.769542 7758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:16.769532 7757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:16.769505 7760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:32:16.770037 7752 server_base.cc:1034] running on GCE node
I20260928 06:32:16.770267 7752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:32:16.770519 7752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:32:16.771709 7752 hybrid_clock.cc:648] HybridClock initialized: now 1790577136771674 us; error 31 us; skew 500 ppm
Sep 28 06:32:16 dist-test-slave-d59m krb5kdc[6307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577136, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:32:16.774288 7752 init.cc:377] Logged in from keytab as kudu/127.22.225.131@KRBTEST.COM (short username kudu)
I20260928 06:32:16.774798 7752 webserver.cc:460] Webserver started at http://127.22.225.131:35703/ using document root <none> and password file <none>
I20260928 06:32:16.775087 7752 fs_manager.cc:359] Metadata directory not provided
I20260928 06:32:16.775136 7752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:32:16.776702 7752 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:32:16.777379 7767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:16.777575 7752 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:32:16.777666 7752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-2/wal
uuid: "80789f481b4948bd9e8732283ec4db11"
format_stamp: "Formatted at 2026-09-28 06:31:35 on dist-test-slave-d59m"
server_key: "ea0c4a6e5e4502802908171df0bbb3e0"
server_key_iv: "65a6afeeee22a63c5edae87189c89117"
server_key_version: "encryptionkey@0"
I20260928 06:32:16.778028 7752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:32:16.804086 7752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:32:16.806591 7752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:32:16.806766 7752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:32:16.807030 7752 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:32:16.807351 7752 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:32:16.807405 7752 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:16.807446 7752 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:32:16.807476 7752 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:16.814028 7752 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.131:46263
I20260928 06:32:16.814054 7880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.131:46263 every 8 connection(s)
I20260928 06:32:16.814507 7752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
I20260928 06:32:16.815722 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:16.815252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:39765 (local address 127.22.225.131:46415)
0928 06:32:16.815442 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:16.815538 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:39765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":92,"threads_started":1}
W20260928 06:32:16.816172 7881 heartbeater.cc:644] Failed to heartbeat to 127.22.225.190:39765 (0 consecutive failures): Network error: Failed to ping master at 127.22.225.190:39765: Client connection negotiation failed: client connection to 127.22.225.190:39765: connect: Connection refused (error 111)
I20260928 06:32:16.816468 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:16.816364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:39765 (local address 127.22.225.131:38713)
0928 06:32:16.816414 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:16.816445 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:39765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 06:32:16.816804 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:16.816714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:39765 (local address 127.22.225.131:55499)
0928 06:32:16.816759 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:16.816787 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:39765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 06:32:16.816959 7881 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 06:32:16.820811 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 7752
I20260928 06:32:17.581936 7885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:17.581663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:39765 (local address 127.22.225.129:43477)
0928 06:32:17.581804 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:17.581887 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:39765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":81,"threads_started":1}
I20260928 06:32:17.676209 7886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:17.675983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:39765 (local address 127.22.225.130:48923)
0928 06:32:17.676128 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:17.676172 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:39765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":109,"threads_started":1}
I20260928 06:32:17.817567 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:17.817305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:39765 (local address 127.22.225.131:46307)
0928 06:32:17.817464 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:17.817499 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:39765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":71,"threads_started":1}
I20260928 06:32:18.066320 7888 raft_consensus.cc:482] T 2f712ca4ff094c479874ee97f0d57a9d P 43a313fadad1418cbcebf59dd0ecb76a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:32:18.066442 7888 raft_consensus.cc:504] T 2f712ca4ff094c479874ee97f0d57a9d P 43a313fadad1418cbcebf59dd0ecb76a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80789f481b4948bd9e8732283ec4db11" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 46263 } } peers { permanent_uuid: "116591f9d0bd4e1590c1d52410720989" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 35187 } } peers { permanent_uuid: "43a313fadad1418cbcebf59dd0ecb76a" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34637 } }
I20260928 06:32:18.066847 7888 leader_election.cc:288] T 2f712ca4ff094c479874ee97f0d57a9d P 43a313fadad1418cbcebf59dd0ecb76a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 80789f481b4948bd9e8732283ec4db11 (127.22.225.131:46263), 116591f9d0bd4e1590c1d52410720989 (127.22.225.130:35187)
Sep 28 06:32:18 dist-test-slave-d59m krb5kdc[6307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577136, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.129@KRBTEST.COM for kudu/127.22.225.130@KRBTEST.COM
Sep 28 06:32:18 dist-test-slave-d59m krb5kdc[6307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577136, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.129@KRBTEST.COM for kudu/127.22.225.131@KRBTEST.COM
I20260928 06:32:18.079798 7885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:18.067036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.131:46263 (local address 127.22.225.129:55969)
0928 06:32:18.067103 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:18.067116 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:32:18.067175 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:32:18.068352 (+ 1177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:32:18.068353 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:32:18.068910 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:32:18.069191 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:18.069199 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:18.071019 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:18.071020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:18.072004 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:18.072007 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:18.072741 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:18.072742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:18.072886 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:18.073551 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:32:18.073562 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:32:18.075703 (+ 2141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:32:18.077942 (+ 2239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:32:18.077950 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:32:18.077952 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:32:18.078391 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:32:18.078710 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:32:18.078710 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:32:18.078711 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:32:18.078808 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:32:18.079301 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:32:18.079306 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:32:18.079416 (+ 110us) client_negotiation.cc:769] Sending connection context
0928 06:32:18.079529 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 06:32:18.079692 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":15}
I20260928 06:32:18.080122 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:18.067042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.130:35187 (local address 127.22.225.129:32771)
0928 06:32:18.067318 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:18.067328 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:32:18.067371 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:32:18.068341 (+ 970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:32:18.068344 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:32:18.068909 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:32:18.069192 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:18.069199 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:18.071406 (+ 2207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:18.071407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:18.072275 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:18.072279 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:18.073035 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:18.073036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:18.073100 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:18.073677 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:32:18.073684 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:32:18.075252 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:32:18.078130 (+ 2878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:32:18.078131 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:32:18.078132 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:32:18.078398 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:32:18.078675 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:32:18.078676 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:32:18.078676 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:32:18.078808 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:32:18.079783 (+ 975us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:32:18.079784 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:32:18.079830 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 06:32:18.079898 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 06:32:18.080001 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":204,"thread_start_us":44,"threads_started":1}
I20260928 06:32:18.079818 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:18.067139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:55969 (local address 127.22.225.131:46263)
0928 06:32:18.067440 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:18.067451 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:18.067489 (+ 38us) server_negotiation.cc:407] Connection header received
0928 06:32:18.068132 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:18.068136 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:18.068188 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:18.068451 (+ 263us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:32:18.069325 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:18.070910 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:18.072105 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:18.072637 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:18.072743 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:18.075807 (+ 3064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:32:18.075810 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:32:18.075811 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:32:18.075821 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:32:18.077837 (+ 2016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:32:18.078490 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:32:18.078493 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:32:18.078494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:32:18.078613 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:32:18.078882 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:32:18.078883 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:32:18.078884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:32:18.079226 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:32:18.079317 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:18.079599 (+ 282us) server_negotiation.cc:299] Negotiation successful
0928 06:32:18.079720 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":55,"threads_started":1}
I20260928 06:32:18.080659 7890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:18.067178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:32771 (local address 127.22.225.130:35187)
0928 06:32:18.067335 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:18.067349 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:18.067377 (+ 28us) server_negotiation.cc:407] Connection header received
0928 06:32:18.068081 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:18.068084 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:18.068132 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:18.068287 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:32:18.070193 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:18.071299 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:18.072371 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:18.072949 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:18.073064 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:18.075361 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:32:18.075366 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:32:18.075366 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:32:18.075379 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:32:18.078038 (+ 2659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:32:18.078492 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:32:18.078495 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:32:18.078496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:32:18.078591 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:32:18.079352 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:32:18.079354 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:32:18.079354 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:32:18.079673 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:32:18.079751 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:18.080288 (+ 537us) server_negotiation.cc:299] Negotiation successful
0928 06:32:18.080482 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":67,"threads_started":1}
W20260928 06:32:18.081295 7502 leader_election.cc:341] T 2f712ca4ff094c479874ee97f0d57a9d P 43a313fadad1418cbcebf59dd0ecb76a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 80789f481b4948bd9e8732283ec4db11 (127.22.225.131:46263): Not found: Tablet not found: 2f712ca4ff094c479874ee97f0d57a9d
W20260928 06:32:18.081350 7500 leader_election.cc:341] T 2f712ca4ff094c479874ee97f0d57a9d P 43a313fadad1418cbcebf59dd0ecb76a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 116591f9d0bd4e1590c1d52410720989 (127.22.225.130:35187): Not found: Tablet not found: 2f712ca4ff094c479874ee97f0d57a9d
I20260928 06:32:18.081403 7502 leader_election.cc:302] T 2f712ca4ff094c479874ee97f0d57a9d P 43a313fadad1418cbcebf59dd0ecb76a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43a313fadad1418cbcebf59dd0ecb76a; no voters: 116591f9d0bd4e1590c1d52410720989, 80789f481b4948bd9e8732283ec4db11
I20260928 06:32:18.081530 7888 raft_consensus.cc:2726] T 2f712ca4ff094c479874ee97f0d57a9d P 43a313fadad1418cbcebf59dd0ecb76a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:32:18.582791 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:18.582480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:39765 (local address 127.22.225.129:50605)
0928 06:32:18.582693 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:18.582744 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:39765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":102,"threads_started":1}
I20260928 06:32:18.677044 7893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:18.676776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:39765 (local address 127.22.225.130:45959)
0928 06:32:18.676950 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:18.676985 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:39765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":118,"threads_started":1}
I20260928 06:32:18.818315 7894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:18.818060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:39765 (local address 127.22.225.131:38233)
0928 06:32:18.818219 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:18.818258 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:39765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":87,"threads_started":1}
I20260928 06:32:19.583429 7895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:19.583216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:39765 (local address 127.22.225.129:48863)
0928 06:32:19.583350 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:19.583383 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:39765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":77,"threads_started":1}
I20260928 06:32:19.677671 7896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:19.677468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:39765 (local address 127.22.225.130:39527)
0928 06:32:19.677589 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:19.677628 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:39765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":76,"threads_started":1}
I20260928 06:32:19.819103 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:19.818865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:39765 (local address 127.22.225.131:39541)
0928 06:32:19.819013 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:19.819056 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:39765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":69,"threads_started":1}
I20260928 06:32:19.940441 7898 raft_consensus.cc:482] T 2f712ca4ff094c479874ee97f0d57a9d P 43a313fadad1418cbcebf59dd0ecb76a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:32:19.940574 7898 raft_consensus.cc:504] T 2f712ca4ff094c479874ee97f0d57a9d P 43a313fadad1418cbcebf59dd0ecb76a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80789f481b4948bd9e8732283ec4db11" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 46263 } } peers { permanent_uuid: "116591f9d0bd4e1590c1d52410720989" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 35187 } } peers { permanent_uuid: "43a313fadad1418cbcebf59dd0ecb76a" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34637 } }
I20260928 06:32:19.941099 7898 leader_election.cc:288] T 2f712ca4ff094c479874ee97f0d57a9d P 43a313fadad1418cbcebf59dd0ecb76a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 80789f481b4948bd9e8732283ec4db11 (127.22.225.131:46263), 116591f9d0bd4e1590c1d52410720989 (127.22.225.130:35187)
W20260928 06:32:19.941180 7502 leader_election.cc:341] T 2f712ca4ff094c479874ee97f0d57a9d P 43a313fadad1418cbcebf59dd0ecb76a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 80789f481b4948bd9e8732283ec4db11 (127.22.225.131:46263): Not found: Tablet not found: 2f712ca4ff094c479874ee97f0d57a9d
W20260928 06:32:19.941358 7500 leader_election.cc:341] T 2f712ca4ff094c479874ee97f0d57a9d P 43a313fadad1418cbcebf59dd0ecb76a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 116591f9d0bd4e1590c1d52410720989 (127.22.225.130:35187): Not found: Tablet not found: 2f712ca4ff094c479874ee97f0d57a9d
I20260928 06:32:19.941425 7500 leader_election.cc:302] T 2f712ca4ff094c479874ee97f0d57a9d P 43a313fadad1418cbcebf59dd0ecb76a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43a313fadad1418cbcebf59dd0ecb76a; no voters: 116591f9d0bd4e1590c1d52410720989, 80789f481b4948bd9e8732283ec4db11
I20260928 06:32:19.941515 7898 raft_consensus.cc:2726] T 2f712ca4ff094c479874ee97f0d57a9d P 43a313fadad1418cbcebf59dd0ecb76a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:32:20.584118 7899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:20.583902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:39765 (local address 127.22.225.129:39653)
0928 06:32:20.584039 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:20.584069 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:39765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":60,"threads_started":1}
I20260928 06:32:20.678444 7900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:20.678197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:39765 (local address 127.22.225.130:58173)
0928 06:32:20.678344 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:20.678384 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:39765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":67,"threads_started":1}
I20260928 06:32:20.819769 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:20.819536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:39765 (local address 127.22.225.131:51597)
0928 06:32:20.819674 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:20.819708 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:39765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":71,"threads_started":1}
I20260928 06:32:21.584937 7902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:21.584651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:39765 (local address 127.22.225.129:39713)
0928 06:32:21.584813 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:21.584869 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:39765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":85,"threads_started":1}
I20260928 06:32:21.679265 7903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:21.679040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:39765 (local address 127.22.225.130:37801)
0928 06:32:21.679169 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:21.679207 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:39765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":77,"threads_started":1}
I20260928 06:32:21.820691 7904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:21.820425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:39765 (local address 127.22.225.131:60585)
0928 06:32:21.820562 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:21.820621 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:39765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":60,"threads_started":1}
I20260928 06:32:22.449803 7905 raft_consensus.cc:482] T 2f712ca4ff094c479874ee97f0d57a9d P 43a313fadad1418cbcebf59dd0ecb76a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:32:22.449885 7905 raft_consensus.cc:504] T 2f712ca4ff094c479874ee97f0d57a9d P 43a313fadad1418cbcebf59dd0ecb76a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80789f481b4948bd9e8732283ec4db11" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 46263 } } peers { permanent_uuid: "116591f9d0bd4e1590c1d52410720989" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 35187 } } peers { permanent_uuid: "43a313fadad1418cbcebf59dd0ecb76a" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34637 } }
I20260928 06:32:22.450088 7905 leader_election.cc:288] T 2f712ca4ff094c479874ee97f0d57a9d P 43a313fadad1418cbcebf59dd0ecb76a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 80789f481b4948bd9e8732283ec4db11 (127.22.225.131:46263), 116591f9d0bd4e1590c1d52410720989 (127.22.225.130:35187)
W20260928 06:32:22.450477 7502 leader_election.cc:341] T 2f712ca4ff094c479874ee97f0d57a9d P 43a313fadad1418cbcebf59dd0ecb76a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 80789f481b4948bd9e8732283ec4db11 (127.22.225.131:46263): Not found: Tablet not found: 2f712ca4ff094c479874ee97f0d57a9d
W20260928 06:32:22.450565 7500 leader_election.cc:341] T 2f712ca4ff094c479874ee97f0d57a9d P 43a313fadad1418cbcebf59dd0ecb76a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 116591f9d0bd4e1590c1d52410720989 (127.22.225.130:35187): Not found: Tablet not found: 2f712ca4ff094c479874ee97f0d57a9d
I20260928 06:32:22.450630 7500 leader_election.cc:302] T 2f712ca4ff094c479874ee97f0d57a9d P 43a313fadad1418cbcebf59dd0ecb76a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43a313fadad1418cbcebf59dd0ecb76a; no voters: 116591f9d0bd4e1590c1d52410720989, 80789f481b4948bd9e8732283ec4db11
I20260928 06:32:22.450727 7905 raft_consensus.cc:2726] T 2f712ca4ff094c479874ee97f0d57a9d P 43a313fadad1418cbcebf59dd0ecb76a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:32:22.585677 7906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:22.585450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:39765 (local address 127.22.225.129:36017)
0928 06:32:22.585591 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:22.585631 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:39765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":75,"threads_started":1}
I20260928 06:32:22.680025 7907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:22.679796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:39765 (local address 127.22.225.130:44257)
0928 06:32:22.679934 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:22.679972 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:39765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":79,"threads_started":1}
I20260928 06:32:22.821408 7908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:22.821211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:39765 (local address 127.22.225.131:54471)
0928 06:32:22.821324 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:22.821364 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:39765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":53,"threads_started":1}
I20260928 06:32:23.586429 7909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:23.586229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:39765 (local address 127.22.225.129:57099)
0928 06:32:23.586357 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:23.586384 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:39765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":56,"threads_started":1}
I20260928 06:32:23.680894 7910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:23.680634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:39765 (local address 127.22.225.130:59355)
0928 06:32:23.680771 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:23.680818 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:39765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":60,"threads_started":1}
I20260928 06:32:23.822101 7911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:23.821889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:39765 (local address 127.22.225.131:51763)
0928 06:32:23.822015 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:23.822051 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:39765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":70,"threads_started":1}
I20260928 06:32:24.587137 7912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:24.586895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:39765 (local address 127.22.225.129:41735)
0928 06:32:24.587053 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:24.587092 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:39765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":71,"threads_started":1}
I20260928 06:32:24.681584 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:24.681364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:39765 (local address 127.22.225.130:52413)
0928 06:32:24.681493 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:24.681534 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:39765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":75,"threads_started":1}
I20260928 06:32:24.822801 7914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:24.822581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:39765 (local address 127.22.225.131:48923)
0928 06:32:24.822715 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:24.822754 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:39765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":91,"threads_started":1}
I20260928 06:32:24.996372 7915 raft_consensus.cc:482] T 2f712ca4ff094c479874ee97f0d57a9d P 43a313fadad1418cbcebf59dd0ecb76a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:32:24.996479 7915 raft_consensus.cc:504] T 2f712ca4ff094c479874ee97f0d57a9d P 43a313fadad1418cbcebf59dd0ecb76a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80789f481b4948bd9e8732283ec4db11" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 46263 } } peers { permanent_uuid: "116591f9d0bd4e1590c1d52410720989" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 35187 } } peers { permanent_uuid: "43a313fadad1418cbcebf59dd0ecb76a" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34637 } }
I20260928 06:32:24.996826 7915 leader_election.cc:288] T 2f712ca4ff094c479874ee97f0d57a9d P 43a313fadad1418cbcebf59dd0ecb76a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 80789f481b4948bd9e8732283ec4db11 (127.22.225.131:46263), 116591f9d0bd4e1590c1d52410720989 (127.22.225.130:35187)
W20260928 06:32:24.997175 7502 leader_election.cc:341] T 2f712ca4ff094c479874ee97f0d57a9d P 43a313fadad1418cbcebf59dd0ecb76a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 80789f481b4948bd9e8732283ec4db11 (127.22.225.131:46263): Not found: Tablet not found: 2f712ca4ff094c479874ee97f0d57a9d
W20260928 06:32:24.997267 7500 leader_election.cc:341] T 2f712ca4ff094c479874ee97f0d57a9d P 43a313fadad1418cbcebf59dd0ecb76a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 116591f9d0bd4e1590c1d52410720989 (127.22.225.130:35187): Not found: Tablet not found: 2f712ca4ff094c479874ee97f0d57a9d
I20260928 06:32:24.997311 7500 leader_election.cc:302] T 2f712ca4ff094c479874ee97f0d57a9d P 43a313fadad1418cbcebf59dd0ecb76a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43a313fadad1418cbcebf59dd0ecb76a; no voters: 116591f9d0bd4e1590c1d52410720989, 80789f481b4948bd9e8732283ec4db11
I20260928 06:32:24.997401 7915 raft_consensus.cc:2726] T 2f712ca4ff094c479874ee97f0d57a9d P 43a313fadad1418cbcebf59dd0ecb76a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:32:25.587790 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:25.587593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:39765 (local address 127.22.225.129:59583)
0928 06:32:25.587714 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:25.587742 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:39765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":59,"threads_started":1}
I20260928 06:32:25.682330 7917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:25.682109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:39765 (local address 127.22.225.130:46841)
0928 06:32:25.682239 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:25.682280 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:39765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":67,"threads_started":1}
I20260928 06:32:25.823529 7918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:25.823311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:39765 (local address 127.22.225.131:51245)
0928 06:32:25.823449 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:25.823482 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:39765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":80,"threads_started":1}
I20260928 06:32:26.588539 7919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:26.588326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:39765 (local address 127.22.225.129:55981)
0928 06:32:26.588465 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:26.588493 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:39765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":75,"threads_started":1}
I20260928 06:32:26.683081 7920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:26.682851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:39765 (local address 127.22.225.130:57433)
0928 06:32:26.682998 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:26.683037 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:39765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":76,"threads_started":1}
I20260928 06:32:26.824329 7921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:26.824067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:39765 (local address 127.22.225.131:59857)
0928 06:32:26.824209 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:26.824259 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:39765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":59,"threads_started":1}
I20260928 06:32:27.589265 7922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:27.589051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:39765 (local address 127.22.225.129:51577)
0928 06:32:27.589181 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:27.589221 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:39765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":85,"threads_started":1}
I20260928 06:32:27.683725 7923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:27.683528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:39765 (local address 127.22.225.130:44291)
0928 06:32:27.683645 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:27.683682 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:39765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":75,"threads_started":1}
I20260928 06:32:27.825138 7924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:27.824908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:39765 (local address 127.22.225.131:35199)
0928 06:32:27.825047 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:27.825092 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:39765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":88,"threads_started":1}
I20260928 06:32:28.590003 7925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:28.589733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:39765 (local address 127.22.225.129:49557)
0928 06:32:28.589877 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:28.589932 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:39765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":54,"threads_started":1}
I20260928 06:32:28.684419 7926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:28.684237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:39765 (local address 127.22.225.130:54651)
0928 06:32:28.684345 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:28.684374 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:39765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":49,"threads_started":1}
I20260928 06:32:28.825876 7927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:28.825628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:39765 (local address 127.22.225.131:58351)
0928 06:32:28.825761 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:28.825809 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:39765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":52,"threads_started":1}
I20260928 06:32:29.590746 7928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:29.590514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:39765 (local address 127.22.225.129:55273)
0928 06:32:29.590658 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:29.590698 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:39765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":80,"threads_started":1}
I20260928 06:32:29.685134 7929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:29.684921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:39765 (local address 127.22.225.130:60559)
0928 06:32:29.685042 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:29.685082 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:39765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":54,"threads_started":1}
I20260928 06:32:29.826713 7930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:29.826435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:39765 (local address 127.22.225.131:33147)
0928 06:32:29.826609 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:29.826650 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:39765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":92,"threads_started":1}
I20260928 06:32:30.591425 7931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:30.591186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:39765 (local address 127.22.225.129:46629)
0928 06:32:30.591312 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:30.591361 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:39765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":70,"threads_started":1}
I20260928 06:32:30.685945 7932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:30.685672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:39765 (local address 127.22.225.130:56791)
0928 06:32:30.685870 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:30.685900 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:39765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":71,"threads_started":1}
I20260928 06:32:30.827518 7933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:30.827242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:39765 (local address 127.22.225.131:47491)
0928 06:32:30.827401 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:30.827436 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:39765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":102,"threads_started":1}
I20260928 06:32:31.013688 7934 raft_consensus.cc:482] T 2f712ca4ff094c479874ee97f0d57a9d P 43a313fadad1418cbcebf59dd0ecb76a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:32:31.013774 7934 raft_consensus.cc:504] T 2f712ca4ff094c479874ee97f0d57a9d P 43a313fadad1418cbcebf59dd0ecb76a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80789f481b4948bd9e8732283ec4db11" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 46263 } } peers { permanent_uuid: "116591f9d0bd4e1590c1d52410720989" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 35187 } } peers { permanent_uuid: "43a313fadad1418cbcebf59dd0ecb76a" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 34637 } }
I20260928 06:32:31.013993 7934 leader_election.cc:288] T 2f712ca4ff094c479874ee97f0d57a9d P 43a313fadad1418cbcebf59dd0ecb76a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 80789f481b4948bd9e8732283ec4db11 (127.22.225.131:46263), 116591f9d0bd4e1590c1d52410720989 (127.22.225.130:35187)
W20260928 06:32:31.014407 7500 leader_election.cc:341] T 2f712ca4ff094c479874ee97f0d57a9d P 43a313fadad1418cbcebf59dd0ecb76a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 116591f9d0bd4e1590c1d52410720989 (127.22.225.130:35187): Not found: Tablet not found: 2f712ca4ff094c479874ee97f0d57a9d
W20260928 06:32:31.014513 7502 leader_election.cc:341] T 2f712ca4ff094c479874ee97f0d57a9d P 43a313fadad1418cbcebf59dd0ecb76a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 80789f481b4948bd9e8732283ec4db11 (127.22.225.131:46263): Not found: Tablet not found: 2f712ca4ff094c479874ee97f0d57a9d
I20260928 06:32:31.014576 7502 leader_election.cc:302] T 2f712ca4ff094c479874ee97f0d57a9d P 43a313fadad1418cbcebf59dd0ecb76a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43a313fadad1418cbcebf59dd0ecb76a; no voters: 116591f9d0bd4e1590c1d52410720989, 80789f481b4948bd9e8732283ec4db11
I20260928 06:32:31.014675 7934 raft_consensus.cc:2726] T 2f712ca4ff094c479874ee97f0d57a9d P 43a313fadad1418cbcebf59dd0ecb76a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:32:31.592284 7935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:31.592072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:39765 (local address 127.22.225.129:56371)
0928 06:32:31.592201 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:31.592238 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:39765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":68,"threads_started":1}
I20260928 06:32:31.686704 7936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:31.686442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:39765 (local address 127.22.225.130:35635)
0928 06:32:31.686587 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:31.686636 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:39765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":50,"threads_started":1}
I20260928 06:32:31.828207 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 7450
I20260928 06:32:31.829170 7937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:31.828106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:39765 (local address 127.22.225.131:45797)
0928 06:32:31.829013 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:31.829051 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.190:39765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":886,"thread_start_us":56,"threads_started":1}
I20260928 06:32:31.833966 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 7617
I20260928 06:32:31.839087 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 7752
2026-09-28T06:32:31Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58629 ms)
[----------] 30 tests from SecurityITest (252702 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:32:31 dist-test-slave-d59m krb5kdc[7940](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:32:31 dist-test-slave-d59m krb5kdc[7940](info): set up 2 sockets
Sep 28 06:32:31 dist-test-slave-d59m krb5kdc[7940](info): commencing operation
krb5kdc: starting...
W20260928 06:32:32.985352 23430 mini_kdc.cc:121] Time spent starting KDC: real 1.124s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Sep 28 06:32:33 dist-test-slave-d59m krb5kdc[7940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577153, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:32:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:32:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab.
Entry for principal kudu/127.22.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab.
I20260928 06:32:33.120270 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:44173
--webserver_interface=127.22.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab
--principal=kudu/127.22.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:36261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.225.190:44173
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:32:33.211371 7953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:32:33.211587 7953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:32:33.211611 7953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:32:33.213220 7953 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:32:33.213289 7953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:32:33.213315 7953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:32:33.213335 7953 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:32:33.213358 7953 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:32:33.215412 7953 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:36261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.225.190:44173
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab
--principal=kudu/127.22.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:44173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:32:33.215714 7953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:32:33.216007 7953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:32:33.221536 7958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:33.221583 7959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:32:33.221701 7953 server_base.cc:1034] running on GCE node
W20260928 06:32:33.221899 7961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:32:33.222162 7953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:32:33.222524 7953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:32:33.225854 7953 hybrid_clock.cc:648] HybridClock initialized: now 1790577153225275 us; error 593 us; skew 500 ppm
Sep 28 06:32:33 dist-test-slave-d59m krb5kdc[7940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577153, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:32:33.235783 7953 init.cc:377] Logged in from keytab as kudu/127.22.225.190@KRBTEST.COM (short username kudu)
I20260928 06:32:33.236318 7953 webserver.cc:460] Webserver started at http://127.22.225.190:42293/ using document root <none> and password file <none>
I20260928 06:32:33.236624 7953 fs_manager.cc:359] Metadata directory not provided
I20260928 06:32:33.236701 7953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:32:33.236848 7953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:32:33.238085 7953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/minicluster-data/master-0/data/instance:
uuid: "63b60bcc9d814a92a94750cff5b45077"
format_stamp: "Formatted at 2026-09-28 06:32:33 on dist-test-slave-d59m"
server_key: "1fec1a96ae1516d0de0a361cdee7a408"
server_key_iv: "0841b68d24305af8ce8e61726b46976d"
server_key_version: "encryptionkey@0"
I20260928 06:32:33.238490 7953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/minicluster-data/master-0/wal/instance:
uuid: "63b60bcc9d814a92a94750cff5b45077"
format_stamp: "Formatted at 2026-09-28 06:32:33 on dist-test-slave-d59m"
server_key: "1fec1a96ae1516d0de0a361cdee7a408"
server_key_iv: "0841b68d24305af8ce8e61726b46976d"
server_key_version: "encryptionkey@0"
I20260928 06:32:33.240439 7953 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:32:33.241403 7968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:33.241608 7953 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:32:33.241731 7953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/minicluster-data/master-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/minicluster-data/master-0/wal
uuid: "63b60bcc9d814a92a94750cff5b45077"
format_stamp: "Formatted at 2026-09-28 06:32:33 on dist-test-slave-d59m"
server_key: "1fec1a96ae1516d0de0a361cdee7a408"
server_key_iv: "0841b68d24305af8ce8e61726b46976d"
server_key_version: "encryptionkey@0"
I20260928 06:32:33.241822 7953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:32:33.280748 7953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:32:33.283224 7953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:32:33.283402 7953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:32:33.288872 7953 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.190:44173
I20260928 06:32:33.288903 8020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.190:44173 every 8 connection(s)
I20260928 06:32:33.289394 7953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
I20260928 06:32:33.290311 8021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:32:33.292979 8021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63b60bcc9d814a92a94750cff5b45077: Bootstrap starting.
I20260928 06:32:33.293638 8021 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63b60bcc9d814a92a94750cff5b45077: Neither blocks nor log segments found. Creating new log.
I20260928 06:32:33.293937 8021 log.cc:824] T 00000000000000000000000000000000 P 63b60bcc9d814a92a94750cff5b45077: Log is configured to *not* fsync() on all Append() calls
I20260928 06:32:33.294741 8021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63b60bcc9d814a92a94750cff5b45077: No bootstrap required, opened a new log
I20260928 06:32:33.296777 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 7953
I20260928 06:32:33.296890 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/minicluster-data/master-0/wal/instance
I20260928 06:32:33.297137 23430 external_mini_cluster.cc:1428] Setting key 35c630bc843f3cfaf4201c36f4cd8e22
I20260928 06:32:33.297784 8021 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63b60bcc9d814a92a94750cff5b45077 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63b60bcc9d814a92a94750cff5b45077" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 44173 } }
I20260928 06:32:33.298000 8021 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63b60bcc9d814a92a94750cff5b45077 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:32:33.298100 8021 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63b60bcc9d814a92a94750cff5b45077 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63b60bcc9d814a92a94750cff5b45077, State: Initialized, Role: FOLLOWER
I20260928 06:32:33.298269 8021 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63b60bcc9d814a92a94750cff5b45077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63b60bcc9d814a92a94750cff5b45077" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 44173 } }
I20260928 06:32:33.298363 8021 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63b60bcc9d814a92a94750cff5b45077 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:32:33.298398 8021 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63b60bcc9d814a92a94750cff5b45077 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:32:33.298480 8021 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63b60bcc9d814a92a94750cff5b45077 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:32:33.299736 8021 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63b60bcc9d814a92a94750cff5b45077 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63b60bcc9d814a92a94750cff5b45077" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 44173 } }
I20260928 06:32:33.299908 8021 leader_election.cc:302] T 00000000000000000000000000000000 P 63b60bcc9d814a92a94750cff5b45077 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63b60bcc9d814a92a94750cff5b45077; no voters:
I20260928 06:32:33.300093 8021 leader_election.cc:288] T 00000000000000000000000000000000 P 63b60bcc9d814a92a94750cff5b45077 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:32:33.300140 8026 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63b60bcc9d814a92a94750cff5b45077 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:32:33.300457 8021 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63b60bcc9d814a92a94750cff5b45077 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:32:33.301043 8026 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63b60bcc9d814a92a94750cff5b45077 [term 1 LEADER]: Becoming Leader. State: Replica: 63b60bcc9d814a92a94750cff5b45077, State: Running, Role: LEADER
I20260928 06:32:33.301160 8026 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63b60bcc9d814a92a94750cff5b45077 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63b60bcc9d814a92a94750cff5b45077" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 44173 } }
I20260928 06:32:33.301538 8026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63b60bcc9d814a92a94750cff5b45077 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63b60bcc9d814a92a94750cff5b45077. Latest consensus state: current_term: 1 leader_uuid: "63b60bcc9d814a92a94750cff5b45077" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63b60bcc9d814a92a94750cff5b45077" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 44173 } } }
I20260928 06:32:33.301977 8026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63b60bcc9d814a92a94750cff5b45077 [sys.catalog]: This master's current role is: LEADER
I20260928 06:32:33.301859 8027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63b60bcc9d814a92a94750cff5b45077 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63b60bcc9d814a92a94750cff5b45077" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63b60bcc9d814a92a94750cff5b45077" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 44173 } } }
I20260928 06:32:33.302196 8027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63b60bcc9d814a92a94750cff5b45077 [sys.catalog]: This master's current role is: LEADER
I20260928 06:32:33.302469 8039 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:32:33.303323 8039 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:32:33.305514 8039 catalog_manager.cc:1347] Generated new cluster ID: c4886715bb6e491e89f320e853676d2b
I20260928 06:32:33.305586 8039 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:32:33 dist-test-slave-d59m krb5kdc[7940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577153, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:32:33.312041 8025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:33.297715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37366 (local address 127.22.225.190:44173)
0928 06:32:33.297910 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:33.297920 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:33.299168 (+ 1248us) server_negotiation.cc:407] Connection header received
0928 06:32:33.299353 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:33.299360 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:33.299428 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:33.299578 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:32:33.300379 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:33.301975 (+ 1596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:33.303272 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:33.303864 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:33.303965 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:33.307372 (+ 3407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:32:33.307377 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:32:33.307377 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:32:33.307389 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:32:33.309789 (+ 2400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:32:33.310250 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:32:33.310253 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:32:33.310254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:32:33.310405 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:32:33.310597 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:32:33.310598 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:32:33.310598 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:32:33.310918 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:32:33.311053 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:33.311478 (+ 425us) server_negotiation.cc:299] Negotiation successful
0928 06:32:33.311619 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":85,"threads_started":1}
I20260928 06:32:33.327538 8039 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:32:33.328174 8039 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:32:33.334920 8039 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 63b60bcc9d814a92a94750cff5b45077: Generated new TSK 0
I20260928 06:32:33.335228 8039 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.225.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab.
Entry for principal kudu/127.22.225.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab.
I20260928 06:32:33.367838 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.129:0
--local_ip_for_outbound_sockets=127.22.225.129
--webserver_interface=127.22.225.129
--webserver_port=0
--tserver_master_addrs=127.22.225.190:44173
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab
--principal=kudu/127.22.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:36261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:32:33.476069 8047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:32:33.476377 8047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:32:33.476423 8047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:32:33.478232 8047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:32:33.478307 8047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:32:33.478354 8047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.129
I20260928 06:32:33.480073 8047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:36261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab
--principal=kudu/127.22.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.225.129
--webserver_port=0
--tserver_master_addrs=127.22.225.190:44173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.129
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:32:33.480347 8047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:32:33.480618 8047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:32:33.486995 8053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:33.487097 8055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:33.487323 8052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:32:33.487473 8047 server_base.cc:1034] running on GCE node
I20260928 06:32:33.487703 8047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:32:33.488000 8047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:32:33.489184 8047 hybrid_clock.cc:648] HybridClock initialized: now 1790577153489161 us; error 36 us; skew 500 ppm
Sep 28 06:32:33 dist-test-slave-d59m krb5kdc[7940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577153, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:32:33.492446 8047 init.cc:377] Logged in from keytab as kudu/127.22.225.129@KRBTEST.COM (short username kudu)
I20260928 06:32:33.493130 8047 webserver.cc:460] Webserver started at http://127.22.225.129:44493/ using document root <none> and password file <none>
I20260928 06:32:33.493455 8047 fs_manager.cc:359] Metadata directory not provided
I20260928 06:32:33.493533 8047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:32:33.493700 8047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:32:33.495062 8047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/minicluster-data/ts-0/data/instance:
uuid: "848a9cec3cce402d9476dc27a2bad7de"
format_stamp: "Formatted at 2026-09-28 06:32:33 on dist-test-slave-d59m"
server_key: "2e443377e1f62af2b9db28f8f64c929c"
server_key_iv: "48d29bb417271ab52cb501fa501d65ab"
server_key_version: "encryptionkey@0"
I20260928 06:32:33.495457 8047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/minicluster-data/ts-0/wal/instance:
uuid: "848a9cec3cce402d9476dc27a2bad7de"
format_stamp: "Formatted at 2026-09-28 06:32:33 on dist-test-slave-d59m"
server_key: "2e443377e1f62af2b9db28f8f64c929c"
server_key_iv: "48d29bb417271ab52cb501fa501d65ab"
server_key_version: "encryptionkey@0"
I20260928 06:32:33.497323 8047 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.003s
I20260928 06:32:33.498333 8062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:33.498562 8047 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:32:33.498668 8047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/minicluster-data/ts-0/wal
uuid: "848a9cec3cce402d9476dc27a2bad7de"
format_stamp: "Formatted at 2026-09-28 06:32:33 on dist-test-slave-d59m"
server_key: "2e443377e1f62af2b9db28f8f64c929c"
server_key_iv: "48d29bb417271ab52cb501fa501d65ab"
server_key_version: "encryptionkey@0"
I20260928 06:32:33.498773 8047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:32:33.528208 8047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:32:33.530910 8047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:32:33.531082 8047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:32:33.531328 8047 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:32:33.531652 8047 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:32:33.531708 8047 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:33.531744 8047 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:32:33.531771 8047 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:33.538403 8047 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.129:46183
I20260928 06:32:33.538437 8175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.129:46183 every 8 connection(s)
I20260928 06:32:33.538911 8047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
I20260928 06:32:33.544389 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 8047
I20260928 06:32:33.544499 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/minicluster-data/ts-0/wal/instance
I20260928 06:32:33.544786 23430 external_mini_cluster.cc:1428] Setting key 046e195dcbdc00d893f102d2dc66b8b6
Sep 28 06:32:33 dist-test-slave-d59m krb5kdc[7940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577153, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.129@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:32:33.549988 8025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:33.539759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:54841 (local address 127.22.225.190:44173)
0928 06:32:33.539810 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:33.539812 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:33.540042 (+ 230us) server_negotiation.cc:407] Connection header received
0928 06:32:33.540617 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:33.540619 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:33.540637 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:33.540697 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:32:33.541798 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:33.542323 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:33.543388 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:33.543661 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:33.543727 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:33.546631 (+ 2904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:32:33.546633 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:32:33.546633 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:32:33.546640 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:32:33.548135 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:32:33.548699 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:32:33.548701 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:32:33.548701 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:32:33.548776 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:32:33.549133 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:32:33.549135 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:32:33.549135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:32:33.549295 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:32:33.549345 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:33.549774 (+ 429us) server_negotiation.cc:299] Negotiation successful
0928 06:32:33.549884 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 06:32:33.550607 8178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:33.539735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:44173 (local address 127.22.225.129:54841)
0928 06:32:33.539913 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:33.539949 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 06:32:33.540470 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:32:33.540778 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:32:33.540782 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:32:33.541396 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:32:33.541688 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:33.541695 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:33.542424 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:33.542425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:33.543297 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:33.543300 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:33.543764 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:33.543765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:33.543917 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:33.544631 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:32:33.544645 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:32:33.546520 (+ 1875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:32:33.548249 (+ 1729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:32:33.548256 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:32:33.548258 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:32:33.548609 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:32:33.548868 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:32:33.548869 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:32:33.548869 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:32:33.549042 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:32:33.549400 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:32:33.549407 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:32:33.549585 (+ 178us) client_negotiation.cc:769] Sending connection context
0928 06:32:33.550121 (+ 536us) client_negotiation.cc:240] Negotiation successful
0928 06:32:33.550322 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":87,"threads_started":1}
I20260928 06:32:33.551155 8176 heartbeater.cc:347] Connected to a master server at 127.22.225.190:44173
I20260928 06:32:33.551263 8176 heartbeater.cc:464] Registering TS with master...
I20260928 06:32:33.551512 8176 heartbeater.cc:511] Master 127.22.225.190:44173 requested a full tablet report, sending...
I20260928 06:32:33.552127 7985 ts_manager.cc:194] Registered new tserver with Master: 848a9cec3cce402d9476dc27a2bad7de (127.22.225.129:46183)
I20260928 06:32:33.553093 7985 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.225.129@KRBTEST.COM'} at 127.22.225.129:54841
WARNING: no policy specified for kudu/127.22.225.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab.
Entry for principal kudu/127.22.225.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab.
I20260928 06:32:33.575332 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.130:0
--local_ip_for_outbound_sockets=127.22.225.130
--webserver_interface=127.22.225.130
--webserver_port=0
--tserver_master_addrs=127.22.225.190:44173
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab
--principal=kudu/127.22.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:36261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:32:33.676393 8181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:32:33.676736 8181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:32:33.676811 8181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:32:33.678035 8181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:32:33.678099 8181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:32:33.678145 8181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.130
I20260928 06:32:33.679698 8181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:36261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab
--principal=kudu/127.22.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.225.130
--webserver_port=0
--tserver_master_addrs=127.22.225.190:44173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.130
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:32:33.679989 8181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:32:33.680307 8181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:32:33.685513 8189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:33.685528 8187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:33.685537 8186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:32:33.685956 8181 server_base.cc:1034] running on GCE node
I20260928 06:32:33.686185 8181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:32:33.686462 8181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:32:33.687624 8181 hybrid_clock.cc:648] HybridClock initialized: now 1790577153687587 us; error 57 us; skew 500 ppm
Sep 28 06:32:33 dist-test-slave-d59m krb5kdc[7940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577153, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:32:33.691102 8181 init.cc:377] Logged in from keytab as kudu/127.22.225.130@KRBTEST.COM (short username kudu)
I20260928 06:32:33.691670 8181 webserver.cc:460] Webserver started at http://127.22.225.130:44355/ using document root <none> and password file <none>
I20260928 06:32:33.691944 8181 fs_manager.cc:359] Metadata directory not provided
I20260928 06:32:33.692015 8181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:32:33.692162 8181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:32:33.693773 8181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/minicluster-data/ts-1/data/instance:
uuid: "5c215a19ee634f89b977d8711da5c5f4"
format_stamp: "Formatted at 2026-09-28 06:32:33 on dist-test-slave-d59m"
server_key: "1c472a4de0acf593472b13604149a32b"
server_key_iv: "5e5e59409430b655c55c29f6a3419a60"
server_key_version: "encryptionkey@0"
I20260928 06:32:33.694226 8181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/minicluster-data/ts-1/wal/instance:
uuid: "5c215a19ee634f89b977d8711da5c5f4"
format_stamp: "Formatted at 2026-09-28 06:32:33 on dist-test-slave-d59m"
server_key: "1c472a4de0acf593472b13604149a32b"
server_key_iv: "5e5e59409430b655c55c29f6a3419a60"
server_key_version: "encryptionkey@0"
I20260928 06:32:33.695896 8181 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:32:33.696972 8196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:33.697162 8181 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:32:33.697260 8181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/minicluster-data/ts-1/wal
uuid: "5c215a19ee634f89b977d8711da5c5f4"
format_stamp: "Formatted at 2026-09-28 06:32:33 on dist-test-slave-d59m"
server_key: "1c472a4de0acf593472b13604149a32b"
server_key_iv: "5e5e59409430b655c55c29f6a3419a60"
server_key_version: "encryptionkey@0"
I20260928 06:32:33.697352 8181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:32:33.726881 8181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:32:33.729836 8181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:32:33.730014 8181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:32:33.730283 8181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:32:33.730643 8181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:32:33.730710 8181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:33.730752 8181 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:32:33.730780 8181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:33.737043 8181 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.130:38989
I20260928 06:32:33.737087 8309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.130:38989 every 8 connection(s)
I20260928 06:32:33.737463 8181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
I20260928 06:32:33.740978 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 8181
I20260928 06:32:33.741168 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/minicluster-data/ts-1/wal/instance
I20260928 06:32:33.741447 23430 external_mini_cluster.cc:1428] Setting key 366d0067ca86dfb96d01394a6b638901
Sep 28 06:32:33 dist-test-slave-d59m krb5kdc[7940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577153, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.130@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:32:33.749193 8025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:33.738436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:56705 (local address 127.22.225.190:44173)
0928 06:32:33.738496 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:33.738498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:33.738732 (+ 234us) server_negotiation.cc:407] Connection header received
0928 06:32:33.739546 (+ 814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:33.739547 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:33.739558 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:33.739633 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:32:33.740896 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:33.741759 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:33.742912 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:33.743117 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:33.743222 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:33.745727 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:32:33.745729 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:32:33.745729 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:32:33.745738 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:32:33.747484 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:32:33.748064 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:32:33.748065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:32:33.748066 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:32:33.748136 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:32:33.748449 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:32:33.748450 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:32:33.748451 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:32:33.748630 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:32:33.748700 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:33.749007 (+ 307us) server_negotiation.cc:299] Negotiation successful
0928 06:32:33.749096 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 06:32:33.749466 8312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:33.738359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:44173 (local address 127.22.225.130:56705)
0928 06:32:33.738577 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:33.738631 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 06:32:33.739427 (+ 796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:32:33.739717 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:32:33.739721 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:32:33.740531 (+ 810us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:32:33.740793 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:33.740798 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:33.741851 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:33.741852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:33.742801 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:33.742804 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:33.743234 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:33.743235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:33.743371 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:33.743942 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:32:33.743952 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:32:33.745594 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:32:33.747595 (+ 2001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:32:33.747603 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:32:33.747606 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:32:33.747959 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:32:33.748227 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:32:33.748228 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:32:33.748228 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:32:33.748370 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:32:33.748701 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:32:33.748709 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:32:33.748854 (+ 145us) client_negotiation.cc:769] Sending connection context
0928 06:32:33.748982 (+ 128us) client_negotiation.cc:240] Negotiation successful
0928 06:32:33.749152 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":87,"threads_started":1}
I20260928 06:32:33.750063 8310 heartbeater.cc:347] Connected to a master server at 127.22.225.190:44173
I20260928 06:32:33.750195 8310 heartbeater.cc:464] Registering TS with master...
I20260928 06:32:33.750401 8310 heartbeater.cc:511] Master 127.22.225.190:44173 requested a full tablet report, sending...
I20260928 06:32:33.750828 7985 ts_manager.cc:194] Registered new tserver with Master: 5c215a19ee634f89b977d8711da5c5f4 (127.22.225.130:38989)
I20260928 06:32:33.751391 7985 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.225.130@KRBTEST.COM'} at 127.22.225.130:56705
WARNING: no policy specified for kudu/127.22.225.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab.
Entry for principal kudu/127.22.225.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab.
I20260928 06:32:33.772027 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.131:0
--local_ip_for_outbound_sockets=127.22.225.131
--webserver_interface=127.22.225.131
--webserver_port=0
--tserver_master_addrs=127.22.225.190:44173
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab
--principal=kudu/127.22.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:36261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:32:33.870942 8315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:32:33.871258 8315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:32:33.871304 8315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:32:33.873049 8315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:32:33.873119 8315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:32:33.873169 8315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.131
I20260928 06:32:33.875442 8315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:36261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab
--principal=kudu/127.22.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.225.131
--webserver_port=0
--tserver_master_addrs=127.22.225.190:44173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.131
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:32:33.875742 8315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:32:33.876014 8315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:32:33.881254 8321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:33.881259 8320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:33.881311 8323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:32:33.881508 8315 server_base.cc:1034] running on GCE node
I20260928 06:32:33.881728 8315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:32:33.882011 8315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:32:33.883196 8315 hybrid_clock.cc:648] HybridClock initialized: now 1790577153883165 us; error 48 us; skew 500 ppm
Sep 28 06:32:33 dist-test-slave-d59m krb5kdc[7940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577153, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:32:33.886127 8315 init.cc:377] Logged in from keytab as kudu/127.22.225.131@KRBTEST.COM (short username kudu)
I20260928 06:32:33.886664 8315 webserver.cc:460] Webserver started at http://127.22.225.131:37389/ using document root <none> and password file <none>
I20260928 06:32:33.886922 8315 fs_manager.cc:359] Metadata directory not provided
I20260928 06:32:33.886996 8315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:32:33.887159 8315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:32:33.888223 8315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/minicluster-data/ts-2/data/instance:
uuid: "6858355257f843bf9896806e9f2be749"
format_stamp: "Formatted at 2026-09-28 06:32:33 on dist-test-slave-d59m"
server_key: "8fe91e208b07c075ba0c91e1e9ffb36c"
server_key_iv: "35e318f77986497aedceb6e9df62bccd"
server_key_version: "encryptionkey@0"
I20260928 06:32:33.888666 8315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/minicluster-data/ts-2/wal/instance:
uuid: "6858355257f843bf9896806e9f2be749"
format_stamp: "Formatted at 2026-09-28 06:32:33 on dist-test-slave-d59m"
server_key: "8fe91e208b07c075ba0c91e1e9ffb36c"
server_key_iv: "35e318f77986497aedceb6e9df62bccd"
server_key_version: "encryptionkey@0"
I20260928 06:32:33.890079 8315 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:32:33.890902 8330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:33.891287 8315 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:32:33.891392 8315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/minicluster-data/ts-2/wal
uuid: "6858355257f843bf9896806e9f2be749"
format_stamp: "Formatted at 2026-09-28 06:32:33 on dist-test-slave-d59m"
server_key: "8fe91e208b07c075ba0c91e1e9ffb36c"
server_key_iv: "35e318f77986497aedceb6e9df62bccd"
server_key_version: "encryptionkey@0"
I20260928 06:32:33.891479 8315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:32:33.930310 8315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:32:33.933058 8315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:32:33.933185 8315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:32:33.933447 8315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:32:33.933816 8315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:32:33.933872 8315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:33.933907 8315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:32:33.933934 8315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:33.941095 8315 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.131:42927
I20260928 06:32:33.941129 8443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.131:42927 every 8 connection(s)
I20260928 06:32:33.941470 8315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
I20260928 06:32:33.948108 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 8315
Sep 28 06:32:33 dist-test-slave-d59m krb5kdc[7940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577153, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.131@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:32:33.948246 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/minicluster-data/ts-2/wal/instance
I20260928 06:32:33.948437 23430 external_mini_cluster.cc:1428] Setting key a5c3340aa12dea5f9026bbcbc3d59946
I20260928 06:32:33.951772 8025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:33.942267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:34015 (local address 127.22.225.190:44173)
0928 06:32:33.942313 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:33.942315 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:33.942536 (+ 221us) server_negotiation.cc:407] Connection header received
0928 06:32:33.943076 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:33.943077 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:33.943089 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:33.943150 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:32:33.944127 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:33.944832 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:33.945937 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:33.946158 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:33.946239 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:33.948785 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:32:33.948786 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:32:33.948787 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:32:33.948795 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:32:33.950319 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:32:33.950819 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:32:33.950819 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:32:33.950820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:32:33.950870 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:32:33.951153 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:32:33.951154 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:32:33.951154 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:32:33.951255 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:32:33.951306 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:33.951638 (+ 332us) server_negotiation.cc:299] Negotiation successful
0928 06:32:33.951699 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:32:33.952059 8446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:33.942236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:44173 (local address 127.22.225.131:34015)
0928 06:32:33.942402 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:33.942441 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 06:32:33.942965 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:32:33.943244 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:32:33.943247 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:32:33.943829 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:32:33.944019 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:33.944024 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:33.944956 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:33.944962 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:33.945843 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:33.945845 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:33.946248 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:33.946249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:33.946373 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:33.946971 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:32:33.946979 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:32:33.948663 (+ 1684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:32:33.950403 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:32:33.950412 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:32:33.950414 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:32:33.950738 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:32:33.950938 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:32:33.950939 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:32:33.950940 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:32:33.951085 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:32:33.951321 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:32:33.951327 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:32:33.951498 (+ 171us) client_negotiation.cc:769] Sending connection context
0928 06:32:33.951620 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 06:32:33.951799 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":89,"threads_started":1}
I20260928 06:32:33.952646 8444 heartbeater.cc:347] Connected to a master server at 127.22.225.190:44173
I20260928 06:32:33.952772 8444 heartbeater.cc:464] Registering TS with master...
I20260928 06:32:33.952952 8444 heartbeater.cc:511] Master 127.22.225.190:44173 requested a full tablet report, sending...
I20260928 06:32:33.953392 7985 ts_manager.cc:194] Registered new tserver with Master: 6858355257f843bf9896806e9f2be749 (127.22.225.131:42927)
I20260928 06:32:33.953778 7985 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.225.131@KRBTEST.COM'} at 127.22.225.131:34015
I20260928 06:32:33.960312 23430 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:32:33.969296 8025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:33.961919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37374 (local address 127.22.225.190:44173)
0928 06:32:33.961989 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:33.961991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:33.962143 (+ 152us) server_negotiation.cc:407] Connection header received
0928 06:32:33.962317 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:33.962319 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:33.962328 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:33.962387 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:32:33.963165 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:33.963641 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:33.964766 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:33.964952 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:33.965011 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:33.966505 (+ 1494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:32:33.966506 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:32:33.966506 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:32:33.966513 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:32:33.968001 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:32:33.968460 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:32:33.968461 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:32:33.968462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:32:33.968523 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:32:33.968788 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:32:33.968789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:32:33.968789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:32:33.968947 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:32:33.969023 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:33.969165 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 06:32:33.969229 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 06:32:33.971293 7985 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37374:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:32:33.973045 7985 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:32:33.980123 8455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:33.975667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.130:38989 (local address 127.0.0.1:32858)
0928 06:32:33.975791 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:33.975834 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:32:33.975905 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:32:33.976172 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:32:33.976176 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:32:33.976187 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:32:33.976276 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:33.976278 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:33.977694 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:33.977696 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:33.979062 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:33.979066 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:33.979775 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:33.979776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:33.979855 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:33.979916 (+ 61us) client_negotiation.cc:769] Sending connection context
0928 06:32:33.979983 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 06:32:33.980036 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"mutex_wait_us":42,"thread_start_us":77,"threads_started":1}
I20260928 06:32:33.980458 8457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:33.975785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32858 (local address 127.22.225.130:38989)
0928 06:32:33.975980 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:33.975986 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:33.975999 (+ 13us) server_negotiation.cc:407] Connection header received
0928 06:32:33.976045 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:33.976047 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:33.976064 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:33.976154 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:32:33.976365 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:33.977566 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:33.979187 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:33.979668 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:33.979773 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:33.979834 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:33.980330 (+ 496us) server_negotiation.cc:299] Negotiation successful
0928 06:32:33.980376 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":76,"threads_started":1}
I20260928 06:32:33.981245 8244 tablet_service.cc:1467] Processing CreateTablet for tablet d2bcfc330f3648b398633051c3719b22 (DEFAULT_TABLE table=test-table [id=f930cf3fefe2462890881a23385d6ff1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:32:33.981614 8244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2bcfc330f3648b398633051c3719b22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:32:33.982174 8459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:33.975927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50822 (local address 127.22.225.131:42927)
0928 06:32:33.976094 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:33.976099 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:33.976228 (+ 129us) server_negotiation.cc:407] Connection header received
0928 06:32:33.976272 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:33.976274 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:33.976290 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:33.976361 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:32:33.976559 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:33.977608 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:33.980993 (+ 3385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:33.981710 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:33.981763 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:33.981814 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:33.982055 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 06:32:33.982098 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":97,"threads_started":1}
I20260928 06:32:33.982425 8458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:33.975691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.131:42927 (local address 127.0.0.1:50822)
0928 06:32:33.976114 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:33.976124 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:32:33.976174 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:32:33.976440 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:32:33.976441 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:32:33.976443 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:32:33.976484 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:33.976485 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:33.977744 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:33.977746 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:33.980892 (+ 3146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:33.980894 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:33.981878 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:33.981879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:33.981956 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:33.981960 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:32:33.982307 (+ 347us) client_negotiation.cc:240] Negotiation successful
0928 06:32:33.982355 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"mutex_wait_us":1982,"thread_start_us":94,"threads_started":1}
I20260928 06:32:33.982782 8456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:33.975915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.129:46183 (local address 127.0.0.1:46120)
0928 06:32:33.976664 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:33.976674 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:32:33.976716 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:32:33.977951 (+ 1235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:32:33.977952 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:32:33.977954 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:32:33.977992 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:33.977993 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:33.979190 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:33.979192 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:33.981059 (+ 1867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:33.981062 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:33.982223 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:33.982224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:33.982443 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:33.982447 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:32:33.982654 (+ 207us) client_negotiation.cc:240] Negotiation successful
0928 06:32:33.982698 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":728,"thread_start_us":74,"threads_started":1}
I20260928 06:32:33.983109 8460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:33.976175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46120 (local address 127.22.225.129:46183)
0928 06:32:33.977792 (+ 1617us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:33.977797 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:33.977806 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:32:33.977842 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:33.977844 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:33.977858 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:33.977936 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:32:33.978059 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:33.979083 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:33.981147 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:33.982055 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:33.982128 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:33.982186 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:33.982974 (+ 788us) server_negotiation.cc:299] Negotiation successful
0928 06:32:33.983024 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1580,"thread_start_us":95,"threads_started":1}
I20260928 06:32:33.983085 8378 tablet_service.cc:1467] Processing CreateTablet for tablet d2bcfc330f3648b398633051c3719b22 (DEFAULT_TABLE table=test-table [id=f930cf3fefe2462890881a23385d6ff1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:32:33.983431 8378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2bcfc330f3648b398633051c3719b22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:32:33.984088 8110 tablet_service.cc:1467] Processing CreateTablet for tablet d2bcfc330f3648b398633051c3719b22 (DEFAULT_TABLE table=test-table [id=f930cf3fefe2462890881a23385d6ff1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:32:33.984396 8110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2bcfc330f3648b398633051c3719b22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:32:33.986399 8462 tablet_bootstrap.cc:492] T d2bcfc330f3648b398633051c3719b22 P 6858355257f843bf9896806e9f2be749: Bootstrap starting.
I20260928 06:32:33.986884 8461 tablet_bootstrap.cc:492] T d2bcfc330f3648b398633051c3719b22 P 5c215a19ee634f89b977d8711da5c5f4: Bootstrap starting.
I20260928 06:32:33.987092 8462 tablet_bootstrap.cc:654] T d2bcfc330f3648b398633051c3719b22 P 6858355257f843bf9896806e9f2be749: Neither blocks nor log segments found. Creating new log.
I20260928 06:32:33.987555 8461 tablet_bootstrap.cc:654] T d2bcfc330f3648b398633051c3719b22 P 5c215a19ee634f89b977d8711da5c5f4: Neither blocks nor log segments found. Creating new log.
I20260928 06:32:33.987720 8462 log.cc:824] T d2bcfc330f3648b398633051c3719b22 P 6858355257f843bf9896806e9f2be749: Log is configured to *not* fsync() on all Append() calls
I20260928 06:32:33.987715 8463 tablet_bootstrap.cc:492] T d2bcfc330f3648b398633051c3719b22 P 848a9cec3cce402d9476dc27a2bad7de: Bootstrap starting.
I20260928 06:32:33.988103 8461 log.cc:824] T d2bcfc330f3648b398633051c3719b22 P 5c215a19ee634f89b977d8711da5c5f4: Log is configured to *not* fsync() on all Append() calls
I20260928 06:32:33.988440 8463 tablet_bootstrap.cc:654] T d2bcfc330f3648b398633051c3719b22 P 848a9cec3cce402d9476dc27a2bad7de: Neither blocks nor log segments found. Creating new log.
I20260928 06:32:33.988531 8462 tablet_bootstrap.cc:492] T d2bcfc330f3648b398633051c3719b22 P 6858355257f843bf9896806e9f2be749: No bootstrap required, opened a new log
I20260928 06:32:33.988626 8462 ts_tablet_manager.cc:1397] T d2bcfc330f3648b398633051c3719b22 P 6858355257f843bf9896806e9f2be749: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:32:33.988958 8461 tablet_bootstrap.cc:492] T d2bcfc330f3648b398633051c3719b22 P 5c215a19ee634f89b977d8711da5c5f4: No bootstrap required, opened a new log
I20260928 06:32:33.989029 8461 ts_tablet_manager.cc:1397] T d2bcfc330f3648b398633051c3719b22 P 5c215a19ee634f89b977d8711da5c5f4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:32:33.989042 8463 log.cc:824] T d2bcfc330f3648b398633051c3719b22 P 848a9cec3cce402d9476dc27a2bad7de: Log is configured to *not* fsync() on all Append() calls
I20260928 06:32:33.989705 8463 tablet_bootstrap.cc:492] T d2bcfc330f3648b398633051c3719b22 P 848a9cec3cce402d9476dc27a2bad7de: No bootstrap required, opened a new log
I20260928 06:32:33.989789 8463 ts_tablet_manager.cc:1397] T d2bcfc330f3648b398633051c3719b22 P 848a9cec3cce402d9476dc27a2bad7de: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 06:32:33.990692 8462 raft_consensus.cc:348] T d2bcfc330f3648b398633051c3719b22 P 6858355257f843bf9896806e9f2be749 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c215a19ee634f89b977d8711da5c5f4" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 38989 } } peers { permanent_uuid: "6858355257f843bf9896806e9f2be749" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 42927 } } peers { permanent_uuid: "848a9cec3cce402d9476dc27a2bad7de" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 46183 } }
I20260928 06:32:33.990881 8462 raft_consensus.cc:374] T d2bcfc330f3648b398633051c3719b22 P 6858355257f843bf9896806e9f2be749 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:32:33.990918 8462 raft_consensus.cc:729] T d2bcfc330f3648b398633051c3719b22 P 6858355257f843bf9896806e9f2be749 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6858355257f843bf9896806e9f2be749, State: Initialized, Role: FOLLOWER
I20260928 06:32:33.991034 8462 consensus_queue.cc:260] T d2bcfc330f3648b398633051c3719b22 P 6858355257f843bf9896806e9f2be749 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c215a19ee634f89b977d8711da5c5f4" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 38989 } } peers { permanent_uuid: "6858355257f843bf9896806e9f2be749" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 42927 } } peers { permanent_uuid: "848a9cec3cce402d9476dc27a2bad7de" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 46183 } }
I20260928 06:32:33.991322 8462 ts_tablet_manager.cc:1428] T d2bcfc330f3648b398633051c3719b22 P 6858355257f843bf9896806e9f2be749: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:32:33.991443 8444 heartbeater.cc:503] Master 127.22.225.190:44173 was elected leader, sending a full tablet report...
I20260928 06:32:33.991324 8463 raft_consensus.cc:348] T d2bcfc330f3648b398633051c3719b22 P 848a9cec3cce402d9476dc27a2bad7de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c215a19ee634f89b977d8711da5c5f4" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 38989 } } peers { permanent_uuid: "6858355257f843bf9896806e9f2be749" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 42927 } } peers { permanent_uuid: "848a9cec3cce402d9476dc27a2bad7de" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 46183 } }
I20260928 06:32:33.991496 8463 raft_consensus.cc:374] T d2bcfc330f3648b398633051c3719b22 P 848a9cec3cce402d9476dc27a2bad7de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:32:33.991544 8463 raft_consensus.cc:729] T d2bcfc330f3648b398633051c3719b22 P 848a9cec3cce402d9476dc27a2bad7de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 848a9cec3cce402d9476dc27a2bad7de, State: Initialized, Role: FOLLOWER
I20260928 06:32:33.991508 8461 raft_consensus.cc:348] T d2bcfc330f3648b398633051c3719b22 P 5c215a19ee634f89b977d8711da5c5f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c215a19ee634f89b977d8711da5c5f4" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 38989 } } peers { permanent_uuid: "6858355257f843bf9896806e9f2be749" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 42927 } } peers { permanent_uuid: "848a9cec3cce402d9476dc27a2bad7de" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 46183 } }
I20260928 06:32:33.991672 8463 consensus_queue.cc:260] T d2bcfc330f3648b398633051c3719b22 P 848a9cec3cce402d9476dc27a2bad7de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c215a19ee634f89b977d8711da5c5f4" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 38989 } } peers { permanent_uuid: "6858355257f843bf9896806e9f2be749" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 42927 } } peers { permanent_uuid: "848a9cec3cce402d9476dc27a2bad7de" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 46183 } }
I20260928 06:32:33.991714 8461 raft_consensus.cc:374] T d2bcfc330f3648b398633051c3719b22 P 5c215a19ee634f89b977d8711da5c5f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:32:33.991770 8461 raft_consensus.cc:729] T d2bcfc330f3648b398633051c3719b22 P 5c215a19ee634f89b977d8711da5c5f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c215a19ee634f89b977d8711da5c5f4, State: Initialized, Role: FOLLOWER
I20260928 06:32:33.991888 8461 consensus_queue.cc:260] T d2bcfc330f3648b398633051c3719b22 P 5c215a19ee634f89b977d8711da5c5f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c215a19ee634f89b977d8711da5c5f4" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 38989 } } peers { permanent_uuid: "6858355257f843bf9896806e9f2be749" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 42927 } } peers { permanent_uuid: "848a9cec3cce402d9476dc27a2bad7de" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 46183 } }
I20260928 06:32:33.991967 8463 ts_tablet_manager.cc:1428] T d2bcfc330f3648b398633051c3719b22 P 848a9cec3cce402d9476dc27a2bad7de: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 06:32:33.992034 8176 heartbeater.cc:503] Master 127.22.225.190:44173 was elected leader, sending a full tablet report...
I20260928 06:32:33.992216 8461 ts_tablet_manager.cc:1428] T d2bcfc330f3648b398633051c3719b22 P 5c215a19ee634f89b977d8711da5c5f4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:32:33.992352 8310 heartbeater.cc:503] Master 127.22.225.190:44173 was elected leader, sending a full tablet report...
W20260928 06:32:34.039757 8177 tablet.cc:2369] T d2bcfc330f3648b398633051c3719b22 P 848a9cec3cce402d9476dc27a2bad7de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:32:34.183652 8469 raft_consensus.cc:482] T d2bcfc330f3648b398633051c3719b22 P 5c215a19ee634f89b977d8711da5c5f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:32:34.183830 8469 raft_consensus.cc:504] T d2bcfc330f3648b398633051c3719b22 P 5c215a19ee634f89b977d8711da5c5f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c215a19ee634f89b977d8711da5c5f4" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 38989 } } peers { permanent_uuid: "6858355257f843bf9896806e9f2be749" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 42927 } } peers { permanent_uuid: "848a9cec3cce402d9476dc27a2bad7de" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 46183 } }
I20260928 06:32:34.184293 8469 leader_election.cc:288] T d2bcfc330f3648b398633051c3719b22 P 5c215a19ee634f89b977d8711da5c5f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6858355257f843bf9896806e9f2be749 (127.22.225.131:42927), 848a9cec3cce402d9476dc27a2bad7de (127.22.225.129:46183)
I20260928 06:32:34.188324 8312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:34.184483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.131:42927 (local address 127.22.225.130:50821)
0928 06:32:34.184548 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:34.184560 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:32:34.184632 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:32:34.184925 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:32:34.184926 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:32:34.184928 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:32:34.184996 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:34.184998 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:34.185942 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:34.185944 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:34.187095 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:34.187098 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:34.188004 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:34.188005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:34.188096 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:34.188100 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:32:34.188172 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 06:32:34.188220 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":4}
I20260928 06:32:34.188674 8459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:34.184717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:50821 (local address 127.22.225.131:42927)
0928 06:32:34.184768 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:34.184771 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:34.184779 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:32:34.184810 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:34.184811 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:34.184820 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:34.184873 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:32:34.185072 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:34.185843 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:34.187205 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:34.187872 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:34.187955 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:34.187966 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:34.188517 (+ 551us) server_negotiation.cc:299] Negotiation successful
0928 06:32:34.188607 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 06:32:34.188966 8398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2bcfc330f3648b398633051c3719b22" candidate_uuid: "5c215a19ee634f89b977d8711da5c5f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6858355257f843bf9896806e9f2be749" is_pre_election: true
I20260928 06:32:34.189100 8398 raft_consensus.cc:2454] T d2bcfc330f3648b398633051c3719b22 P 6858355257f843bf9896806e9f2be749 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c215a19ee634f89b977d8711da5c5f4 in term 0.
I20260928 06:32:34.189352 8199 leader_election.cc:302] T d2bcfc330f3648b398633051c3719b22 P 5c215a19ee634f89b977d8711da5c5f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c215a19ee634f89b977d8711da5c5f4, 6858355257f843bf9896806e9f2be749; no voters:
I20260928 06:32:34.189482 8469 raft_consensus.cc:2781] T d2bcfc330f3648b398633051c3719b22 P 5c215a19ee634f89b977d8711da5c5f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:32:34.189554 8469 raft_consensus.cc:482] T d2bcfc330f3648b398633051c3719b22 P 5c215a19ee634f89b977d8711da5c5f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:32:34.189604 8469 raft_consensus.cc:3037] T d2bcfc330f3648b398633051c3719b22 P 5c215a19ee634f89b977d8711da5c5f4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:32:34.189836 8460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:34.184605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:40243 (local address 127.22.225.129:46183)
0928 06:32:34.184653 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:34.184656 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:34.184841 (+ 185us) server_negotiation.cc:407] Connection header received
0928 06:32:34.184944 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:34.184945 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:34.184958 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:34.184990 (+ 32us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:32:34.185202 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:34.185890 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:34.188619 (+ 2729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:34.189346 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:34.189434 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:34.189445 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:34.189664 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 06:32:34.189733 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 06:32:34.190116 8470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:34.184483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.129:46183 (local address 127.22.225.130:40243)
0928 06:32:34.184779 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:34.184790 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:32:34.184866 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:32:34.185059 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:32:34.185059 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:32:34.185061 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:32:34.185112 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:34.185113 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:34.185993 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:34.185994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:34.187272 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:34.187276 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:34.189488 (+ 2212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:34.189489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:34.189573 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:34.189577 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:32:34.189940 (+ 363us) client_negotiation.cc:240] Negotiation successful
0928 06:32:34.189990 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"mutex_wait_us":14,"thread_start_us":126,"threads_started":1}
I20260928 06:32:34.190521 8130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2bcfc330f3648b398633051c3719b22" candidate_uuid: "5c215a19ee634f89b977d8711da5c5f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "848a9cec3cce402d9476dc27a2bad7de" is_pre_election: true
I20260928 06:32:34.190663 8469 raft_consensus.cc:504] T d2bcfc330f3648b398633051c3719b22 P 5c215a19ee634f89b977d8711da5c5f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c215a19ee634f89b977d8711da5c5f4" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 38989 } } peers { permanent_uuid: "6858355257f843bf9896806e9f2be749" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 42927 } } peers { permanent_uuid: "848a9cec3cce402d9476dc27a2bad7de" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 46183 } }
I20260928 06:32:34.190726 8130 raft_consensus.cc:2454] T d2bcfc330f3648b398633051c3719b22 P 848a9cec3cce402d9476dc27a2bad7de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c215a19ee634f89b977d8711da5c5f4 in term 0.
I20260928 06:32:34.190933 8469 leader_election.cc:288] T d2bcfc330f3648b398633051c3719b22 P 5c215a19ee634f89b977d8711da5c5f4 [CANDIDATE]: Term 1 election: Requested vote from peers 6858355257f843bf9896806e9f2be749 (127.22.225.131:42927), 848a9cec3cce402d9476dc27a2bad7de (127.22.225.129:46183)
I20260928 06:32:34.191010 8130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2bcfc330f3648b398633051c3719b22" candidate_uuid: "5c215a19ee634f89b977d8711da5c5f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "848a9cec3cce402d9476dc27a2bad7de"
I20260928 06:32:34.191094 8130 raft_consensus.cc:3037] T d2bcfc330f3648b398633051c3719b22 P 848a9cec3cce402d9476dc27a2bad7de [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:32:34.191095 8398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2bcfc330f3648b398633051c3719b22" candidate_uuid: "5c215a19ee634f89b977d8711da5c5f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6858355257f843bf9896806e9f2be749"
I20260928 06:32:34.191191 8398 raft_consensus.cc:3037] T d2bcfc330f3648b398633051c3719b22 P 6858355257f843bf9896806e9f2be749 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:32:34.191856 8130 raft_consensus.cc:2454] T d2bcfc330f3648b398633051c3719b22 P 848a9cec3cce402d9476dc27a2bad7de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c215a19ee634f89b977d8711da5c5f4 in term 1.
I20260928 06:32:34.192060 8198 leader_election.cc:302] T d2bcfc330f3648b398633051c3719b22 P 5c215a19ee634f89b977d8711da5c5f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c215a19ee634f89b977d8711da5c5f4, 848a9cec3cce402d9476dc27a2bad7de; no voters:
W20260928 06:32:34.192157 8445 tablet.cc:2369] T d2bcfc330f3648b398633051c3719b22 P 6858355257f843bf9896806e9f2be749: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:32:34.192196 8469 raft_consensus.cc:2781] T d2bcfc330f3648b398633051c3719b22 P 5c215a19ee634f89b977d8711da5c5f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:32:34.192322 8469 raft_consensus.cc:686] T d2bcfc330f3648b398633051c3719b22 P 5c215a19ee634f89b977d8711da5c5f4 [term 1 LEADER]: Becoming Leader. State: Replica: 5c215a19ee634f89b977d8711da5c5f4, State: Running, Role: LEADER
I20260928 06:32:34.192370 8398 raft_consensus.cc:2454] T d2bcfc330f3648b398633051c3719b22 P 6858355257f843bf9896806e9f2be749 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c215a19ee634f89b977d8711da5c5f4 in term 1.
I20260928 06:32:34.192404 8469 consensus_queue.cc:237] T d2bcfc330f3648b398633051c3719b22 P 5c215a19ee634f89b977d8711da5c5f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c215a19ee634f89b977d8711da5c5f4" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 38989 } } peers { permanent_uuid: "6858355257f843bf9896806e9f2be749" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 42927 } } peers { permanent_uuid: "848a9cec3cce402d9476dc27a2bad7de" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 46183 } }
I20260928 06:32:34.193430 7984 catalog_manager.cc:5495] T d2bcfc330f3648b398633051c3719b22 P 5c215a19ee634f89b977d8711da5c5f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5c215a19ee634f89b977d8711da5c5f4 (127.22.225.130). New cstate: current_term: 1 leader_uuid: "5c215a19ee634f89b977d8711da5c5f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c215a19ee634f89b977d8711da5c5f4" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 38989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6858355257f843bf9896806e9f2be749" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 42927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "848a9cec3cce402d9476dc27a2bad7de" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 46183 } health_report { overall_health: UNKNOWN } } }
I20260928 06:32:34.211625 8457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:34.207347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32864 (local address 127.22.225.130:38989)
0928 06:32:34.207398 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:34.207400 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:34.208766 (+ 1366us) server_negotiation.cc:407] Connection header received
0928 06:32:34.208874 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:34.208875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:34.208903 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:34.208996 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:32:34.209169 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:34.209640 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:34.210593 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:34.210900 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:34.210957 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:34.211148 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:32:34.211304 (+ 156us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:32:34.211386 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:34.211488 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 06:32:34.211513 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 06:32:34.213865 8130 raft_consensus.cc:1261] T d2bcfc330f3648b398633051c3719b22 P 848a9cec3cce402d9476dc27a2bad7de [term 1 FOLLOWER]: Refusing update from remote peer 5c215a19ee634f89b977d8711da5c5f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:32:34.213888 8398 raft_consensus.cc:1261] T d2bcfc330f3648b398633051c3719b22 P 6858355257f843bf9896806e9f2be749 [term 1 FOLLOWER]: Refusing update from remote peer 5c215a19ee634f89b977d8711da5c5f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:32:34.214279 8471 consensus_queue.cc:1035] T d2bcfc330f3648b398633051c3719b22 P 5c215a19ee634f89b977d8711da5c5f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "848a9cec3cce402d9476dc27a2bad7de" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 46183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:32:34.214448 8469 consensus_queue.cc:1035] T d2bcfc330f3648b398633051c3719b22 P 5c215a19ee634f89b977d8711da5c5f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6858355257f843bf9896806e9f2be749" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 42927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:32:34.216859 8474 mvcc.cc:205] Tried to move back new op lower bound from 7334204023656722432 to 7334204023572656128. Current Snapshot: MvccSnapshot[applied={T|T < 7334204023656722432}]
I20260928 06:32:34.217464 8475 mvcc.cc:205] Tried to move back new op lower bound from 7334204023656722432 to 7334204023572656128. Current Snapshot: MvccSnapshot[applied={T|T < 7334204023656722432}]
I20260928 06:32:34.220029 7983 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37374:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:32:34.220134 7983 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37374:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:32:34.220938 7983 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 63b60bcc9d814a92a94750cff5b45077: Sending DeleteTablet for 3 replicas of tablet d2bcfc330f3648b398633051c3719b22
I20260928 06:32:34.221321 8244 tablet_service.cc:1514] Processing DeleteTablet for tablet d2bcfc330f3648b398633051c3719b22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:32:34 UTC) from {username='kudu', principal='kudu/127.22.225.190@KRBTEST.COM'} at 127.0.0.1:32858
I20260928 06:32:34.221366 8378 tablet_service.cc:1514] Processing DeleteTablet for tablet d2bcfc330f3648b398633051c3719b22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:32:34 UTC) from {username='kudu', principal='kudu/127.22.225.190@KRBTEST.COM'} at 127.0.0.1:50822
I20260928 06:32:34.221601 8484 tablet_replica.cc:330] T d2bcfc330f3648b398633051c3719b22 P 6858355257f843bf9896806e9f2be749: stopping tablet replica
I20260928 06:32:34.221570 8110 tablet_service.cc:1514] Processing DeleteTablet for tablet d2bcfc330f3648b398633051c3719b22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:32:34 UTC) from {username='kudu', principal='kudu/127.22.225.190@KRBTEST.COM'} at 127.0.0.1:46120
I20260928 06:32:34.221781 8483 tablet_replica.cc:330] T d2bcfc330f3648b398633051c3719b22 P 5c215a19ee634f89b977d8711da5c5f4: stopping tablet replica
I20260928 06:32:34.221864 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 8047
I20260928 06:32:34.221908 8483 raft_consensus.cc:2229] T d2bcfc330f3648b398633051c3719b22 P 5c215a19ee634f89b977d8711da5c5f4 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:32:34.223120 8483 raft_consensus.cc:2258] T d2bcfc330f3648b398633051c3719b22 P 5c215a19ee634f89b977d8711da5c5f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:32:34.223630 8483 ts_tablet_manager.cc:1905] T d2bcfc330f3648b398633051c3719b22 P 5c215a19ee634f89b977d8711da5c5f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:32:34.224943 8484 raft_consensus.cc:2229] T d2bcfc330f3648b398633051c3719b22 P 6858355257f843bf9896806e9f2be749 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:32:34.225086 8484 raft_consensus.cc:2258] T d2bcfc330f3648b398633051c3719b22 P 6858355257f843bf9896806e9f2be749 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:32:34.225562 8484 ts_tablet_manager.cc:1905] T d2bcfc330f3648b398633051c3719b22 P 6858355257f843bf9896806e9f2be749: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:32:34.225996 8483 ts_tablet_manager.cc:1918] T d2bcfc330f3648b398633051c3719b22 P 5c215a19ee634f89b977d8711da5c5f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:32:34.226117 8483 log.cc:1196] T d2bcfc330f3648b398633051c3719b22 P 5c215a19ee634f89b977d8711da5c5f4: Deleting WAL directory at /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/minicluster-data/ts-1/wal/wals/d2bcfc330f3648b398633051c3719b22
I20260928 06:32:34.226449 8483 ts_tablet_manager.cc:1939] T d2bcfc330f3648b398633051c3719b22 P 5c215a19ee634f89b977d8711da5c5f4: Deleting consensus metadata
I20260928 06:32:34.226904 7972 catalog_manager.cc:4841] TS 5c215a19ee634f89b977d8711da5c5f4 (127.22.225.130:38989): tablet d2bcfc330f3648b398633051c3719b22 (table test-table [id=f930cf3fefe2462890881a23385d6ff1]) successfully deleted
I20260928 06:32:34.227167 8484 ts_tablet_manager.cc:1918] T d2bcfc330f3648b398633051c3719b22 P 6858355257f843bf9896806e9f2be749: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:32:34.227244 8484 log.cc:1196] T d2bcfc330f3648b398633051c3719b22 P 6858355257f843bf9896806e9f2be749: Deleting WAL directory at /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576899143097-23430-0/minicluster-data/ts-2/wal/wals/d2bcfc330f3648b398633051c3719b22
I20260928 06:32:34.227522 8484 ts_tablet_manager.cc:1939] T d2bcfc330f3648b398633051c3719b22 P 6858355257f843bf9896806e9f2be749: Deleting consensus metadata
I20260928 06:32:34.227946 7971 catalog_manager.cc:4841] TS 6858355257f843bf9896806e9f2be749 (127.22.225.131:42927): tablet d2bcfc330f3648b398633051c3719b22 (table test-table [id=f930cf3fefe2462890881a23385d6ff1]) successfully deleted
W20260928 06:32:34.229226 7970 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.225.129:46183 (error 108)
I20260928 06:32:34.229754 8456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:34.229637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.129:46183 (local address 127.0.0.1:46124)
0928 06:32:34.229676 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:34.229715 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.129:46183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260928 06:32:34.230006 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 8181
W20260928 06:32:34.230144 7970 catalog_manager.cc:4584] TS 848a9cec3cce402d9476dc27a2bad7de (127.22.225.129:46183): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d2bcfc330f3648b398633051c3719b22: Network error: Client connection negotiation failed: client connection to 127.22.225.129:46183: connect: Connection refused (error 111)
I20260928 06:32:34.236327 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 8315
I20260928 06:32:34.242647 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 7953
2026-09-28T06:32:34Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2405 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T06:32:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:32:34Z Disabled control of system clock
I20260928 06:32:34.275727 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576899143097-23430-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576899143097-23430-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576899143097-23430-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:33453
--webserver_interface=127.22.225.190
--webserver_port=0
--builtin_ntp_servers=127.22.225.148:36625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.225.190:33453
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 06:32:34.380473 8490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:32:34.380751 8490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:32:34.380776 8490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:32:34.381942 8490 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:32:34.381981 8490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:32:34.381994 8490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:32:34.382006 8490 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:32:34.382052 8490 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:32:34.383370 8490 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:36625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576899143097-23430-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576899143097-23430-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.225.190:33453
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:33453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576899143097-23430-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:32:34.383630 8490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:32:34.383939 8490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:32:34.388823 8496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:34.388821 8495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:32:34.389070 8490 server_base.cc:1034] running on GCE node
W20260928 06:32:34.388837 8498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:32:34.389477 8490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:32:34.389799 8490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:32:34.390993 8490 hybrid_clock.cc:648] HybridClock initialized: now 1790577154390969 us; error 49 us; skew 500 ppm
I20260928 06:32:34.392388 8490 webserver.cc:460] Webserver started at http://127.22.225.190:43409/ using document root <none> and password file <none>
I20260928 06:32:34.392668 8490 fs_manager.cc:359] Metadata directory not provided
I20260928 06:32:34.392725 8490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:32:34.392861 8490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:32:34.394250 8490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576899143097-23430-0/minicluster-data/master-0/data/instance:
uuid: "9bed558d95154b8daa90c0e990412ceb"
format_stamp: "Formatted at 2026-09-28 06:32:34 on dist-test-slave-d59m"
server_key: "5b20945d9e5fb46f13f557ab9b281e42"
server_key_iv: "96adb9ba57e6a0b2ce13201eda736285"
server_key_version: "encryptionkey@0"
I20260928 06:32:34.394649 8490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576899143097-23430-0/minicluster-data/master-0/wal/instance:
uuid: "9bed558d95154b8daa90c0e990412ceb"
format_stamp: "Formatted at 2026-09-28 06:32:34 on dist-test-slave-d59m"
server_key: "5b20945d9e5fb46f13f557ab9b281e42"
server_key_iv: "96adb9ba57e6a0b2ce13201eda736285"
server_key_version: "encryptionkey@0"
I20260928 06:32:34.396219 8490 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:32:34.397119 8504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:34.397320 8490 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:32:34.397411 8490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576899143097-23430-0/minicluster-data/master-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576899143097-23430-0/minicluster-data/master-0/wal
uuid: "9bed558d95154b8daa90c0e990412ceb"
format_stamp: "Formatted at 2026-09-28 06:32:34 on dist-test-slave-d59m"
server_key: "5b20945d9e5fb46f13f557ab9b281e42"
server_key_iv: "96adb9ba57e6a0b2ce13201eda736285"
server_key_version: "encryptionkey@0"
I20260928 06:32:34.397480 8490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576899143097-23430-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576899143097-23430-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576899143097-23430-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:32:34.448557 8490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:32:34.448962 8490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:32:34.449105 8490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:32:34.453819 8490 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.190:33453
I20260928 06:32:34.453845 8556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.190:33453 every 8 connection(s)
I20260928 06:32:34.454244 8490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
I20260928 06:32:34.455268 8557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:32:34.457876 8557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bed558d95154b8daa90c0e990412ceb: Bootstrap starting.
I20260928 06:32:34.458632 8557 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9bed558d95154b8daa90c0e990412ceb: Neither blocks nor log segments found. Creating new log.
I20260928 06:32:34.458933 8557 log.cc:824] T 00000000000000000000000000000000 P 9bed558d95154b8daa90c0e990412ceb: Log is configured to *not* fsync() on all Append() calls
I20260928 06:32:34.459631 8557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bed558d95154b8daa90c0e990412ceb: No bootstrap required, opened a new log
I20260928 06:32:34.461506 8557 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9bed558d95154b8daa90c0e990412ceb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bed558d95154b8daa90c0e990412ceb" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 33453 } }
I20260928 06:32:34.461670 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 8490
I20260928 06:32:34.461704 8557 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9bed558d95154b8daa90c0e990412ceb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:32:34.461748 8557 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9bed558d95154b8daa90c0e990412ceb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bed558d95154b8daa90c0e990412ceb, State: Initialized, Role: FOLLOWER
I20260928 06:32:34.461774 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576899143097-23430-0/minicluster-data/master-0/wal/instance
I20260928 06:32:34.462015 23430 external_mini_cluster.cc:1428] Setting key 710abe77b4759e4539df7d81b1023468
I20260928 06:32:34.461987 8557 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9bed558d95154b8daa90c0e990412ceb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bed558d95154b8daa90c0e990412ceb" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 33453 } }
I20260928 06:32:34.462069 8557 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9bed558d95154b8daa90c0e990412ceb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:32:34.462160 8557 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9bed558d95154b8daa90c0e990412ceb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:32:34.462226 8557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9bed558d95154b8daa90c0e990412ceb [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:32:34.462932 8557 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9bed558d95154b8daa90c0e990412ceb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bed558d95154b8daa90c0e990412ceb" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 33453 } }
I20260928 06:32:34.463053 8557 leader_election.cc:302] T 00000000000000000000000000000000 P 9bed558d95154b8daa90c0e990412ceb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9bed558d95154b8daa90c0e990412ceb; no voters:
I20260928 06:32:34.463250 8557 leader_election.cc:288] T 00000000000000000000000000000000 P 9bed558d95154b8daa90c0e990412ceb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:32:34.463279 8562 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9bed558d95154b8daa90c0e990412ceb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:32:34.463388 8562 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9bed558d95154b8daa90c0e990412ceb [term 1 LEADER]: Becoming Leader. State: Replica: 9bed558d95154b8daa90c0e990412ceb, State: Running, Role: LEADER
I20260928 06:32:34.463536 8557 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9bed558d95154b8daa90c0e990412ceb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:32:34.463536 8562 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9bed558d95154b8daa90c0e990412ceb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bed558d95154b8daa90c0e990412ceb" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 33453 } }
I20260928 06:32:34.465425 8562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bed558d95154b8daa90c0e990412ceb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9bed558d95154b8daa90c0e990412ceb. Latest consensus state: current_term: 1 leader_uuid: "9bed558d95154b8daa90c0e990412ceb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bed558d95154b8daa90c0e990412ceb" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 33453 } } }
I20260928 06:32:34.465567 8562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bed558d95154b8daa90c0e990412ceb [sys.catalog]: This master's current role is: LEADER
I20260928 06:32:34.465610 8563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bed558d95154b8daa90c0e990412ceb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9bed558d95154b8daa90c0e990412ceb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bed558d95154b8daa90c0e990412ceb" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 33453 } } }
I20260928 06:32:34.465699 8563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bed558d95154b8daa90c0e990412ceb [sys.catalog]: This master's current role is: LEADER
I20260928 06:32:34.465859 8576 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:32:34.466586 8576 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:32:34.468307 8576 catalog_manager.cc:1347] Generated new cluster ID: 4e1764cab2d6486bb1ecbe96edebbc26
I20260928 06:32:34.468361 8576 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:32:34.469309 8561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:34.462565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57874 (local address 127.22.225.190:33453)
0928 06:32:34.462807 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:34.462817 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:34.463720 (+ 903us) server_negotiation.cc:407] Connection header received
0928 06:32:34.463940 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:34.463947 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:34.464030 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:34.464168 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:32:34.464510 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:34.465992 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:34.467142 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:34.467617 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:34.467764 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:34.468480 (+ 716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:32:34.468484 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:32:34.468484 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:32:34.468742 (+ 258us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:32:34.468784 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:32:34.468799 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:32:34.468852 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:34.469026 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 06:32:34.469099 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":99,"threads_started":1}
I20260928 06:32:34.482542 8576 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:32:34.483300 8576 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:32:34.488116 8576 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9bed558d95154b8daa90c0e990412ceb: Generated new TSK 0
I20260928 06:32:34.488325 8576 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:32:34.491250 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576899143097-23430-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576899143097-23430-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576899143097-23430-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.129:0
--local_ip_for_outbound_sockets=127.22.225.129
--webserver_interface=127.22.225.129
--webserver_port=0
--tserver_master_addrs=127.22.225.190:33453
--builtin_ntp_servers=127.22.225.148:36625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 06:32:34.611469 8581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:32:34.611780 8581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:32:34.611825 8581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:32:34.613615 8581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:32:34.613688 8581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:32:34.613732 8581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.129
I20260928 06:32:34.615841 8581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:36625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576899143097-23430-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576899143097-23430-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.225.129
--webserver_port=0
--tserver_master_addrs=127.22.225.190:33453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.129
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576899143097-23430-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:32:34.616124 8581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:32:34.616539 8581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:32:34.622871 8586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:34.622895 8589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:34.623270 8587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:32:34.624279 8581 server_base.cc:1034] running on GCE node
I20260928 06:32:34.624544 8581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:32:34.624882 8581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:32:34.626071 8581 hybrid_clock.cc:648] HybridClock initialized: now 1790577154626046 us; error 37 us; skew 500 ppm
I20260928 06:32:34.627719 8581 webserver.cc:460] Webserver started at http://127.22.225.129:41671/ using document root <none> and password file <none>
I20260928 06:32:34.628010 8581 fs_manager.cc:359] Metadata directory not provided
I20260928 06:32:34.628067 8581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:32:34.628206 8581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:32:34.629662 8581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576899143097-23430-0/minicluster-data/ts-0/data/instance:
uuid: "c5ecb1a8717342febf88ac86b45ebe2f"
format_stamp: "Formatted at 2026-09-28 06:32:34 on dist-test-slave-d59m"
server_key: "c0256301734c765c3c583c982b45dbb0"
server_key_iv: "d9023942a7e4a7fb37a4789cedbac3a5"
server_key_version: "encryptionkey@0"
I20260928 06:32:34.630065 8581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576899143097-23430-0/minicluster-data/ts-0/wal/instance:
uuid: "c5ecb1a8717342febf88ac86b45ebe2f"
format_stamp: "Formatted at 2026-09-28 06:32:34 on dist-test-slave-d59m"
server_key: "c0256301734c765c3c583c982b45dbb0"
server_key_iv: "d9023942a7e4a7fb37a4789cedbac3a5"
server_key_version: "encryptionkey@0"
I20260928 06:32:34.631852 8581 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:32:34.632777 8595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:34.632952 8581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:32:34.633029 8581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576899143097-23430-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576899143097-23430-0/minicluster-data/ts-0/wal
uuid: "c5ecb1a8717342febf88ac86b45ebe2f"
format_stamp: "Formatted at 2026-09-28 06:32:34 on dist-test-slave-d59m"
server_key: "c0256301734c765c3c583c982b45dbb0"
server_key_iv: "d9023942a7e4a7fb37a4789cedbac3a5"
server_key_version: "encryptionkey@0"
I20260928 06:32:34.633101 8581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576899143097-23430-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576899143097-23430-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576899143097-23430-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:32:34.643867 8581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:32:34.644268 8581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:32:34.644404 8581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:32:34.644676 8581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:32:34.645078 8581 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:32:34.645120 8581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:34.645151 8581 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:32:34.645206 8581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:34.652267 8581 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.129:40789
I20260928 06:32:34.652314 8708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.129:40789 every 8 connection(s)
I20260928 06:32:34.652760 8581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
I20260928 06:32:34.657436 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 8581
I20260928 06:32:34.657558 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576899143097-23430-0/minicluster-data/ts-0/wal/instance
I20260928 06:32:34.657769 23430 external_mini_cluster.cc:1428] Setting key ea0f492b59665c76167216b2016ff19a
I20260928 06:32:34.658860 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576899143097-23430-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576899143097-23430-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576899143097-23430-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.130:0
--local_ip_for_outbound_sockets=127.22.225.130
--webserver_interface=127.22.225.130
--webserver_port=0
--tserver_master_addrs=127.22.225.190:33453
--builtin_ntp_servers=127.22.225.148:36625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 06:32:34.659355 8561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:34.653670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:59203 (local address 127.22.225.190:33453)
0928 06:32:34.653722 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:34.653724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:34.654020 (+ 296us) server_negotiation.cc:407] Connection header received
0928 06:32:34.654714 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:34.654715 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:34.654727 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:34.654779 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:32:34.655650 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:34.656140 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:34.657648 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:34.657951 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:34.657991 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:34.659063 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:32:34.659065 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:32:34.659065 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:32:34.659070 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:32:34.659085 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:32:34.659090 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:32:34.659116 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:34.659247 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 06:32:34.659274 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:32:34.659788 8711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:34.653666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:33453 (local address 127.22.225.129:59203)
0928 06:32:34.653880 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:34.653926 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 06:32:34.654574 (+ 648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:32:34.654859 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:32:34.654863 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:32:34.654873 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:32:34.655539 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:34.655546 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:34.656233 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:34.656234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:34.657540 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:34.657543 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:34.658077 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:34.658078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:34.658239 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:34.658896 (+ 657us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:32:34.658896 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:32:34.658937 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:32:34.658964 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:32:34.659153 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:32:34.659157 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:32:34.659160 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:32:34.659469 (+ 309us) client_negotiation.cc:240] Negotiation successful
0928 06:32:34.659577 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":96,"threads_started":1}
I20260928 06:32:34.660331 8709 heartbeater.cc:347] Connected to a master server at 127.22.225.190:33453
I20260928 06:32:34.660431 8709 heartbeater.cc:464] Registering TS with master...
I20260928 06:32:34.660672 8709 heartbeater.cc:511] Master 127.22.225.190:33453 requested a full tablet report, sending...
I20260928 06:32:34.661257 8521 ts_manager.cc:194] Registered new tserver with Master: c5ecb1a8717342febf88ac86b45ebe2f (127.22.225.129:40789)
I20260928 06:32:34.662150 8521 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.225.129:59203
W20260928 06:32:34.749579 8712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:32:34.749804 8712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:32:34.749840 8712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:32:34.751017 8712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:32:34.751080 8712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:32:34.751122 8712 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.130
I20260928 06:32:34.752732 8712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:36625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576899143097-23430-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576899143097-23430-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.225.130
--webserver_port=0
--tserver_master_addrs=127.22.225.190:33453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8712
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.130
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576899143097-23430-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:32:34.752995 8712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:32:34.753314 8712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:32:34.759315 8718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:34.759394 8720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:34.759565 8717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:32:34.759809 8712 server_base.cc:1034] running on GCE node
I20260928 06:32:34.760028 8712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:32:34.760308 8712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:32:34.761457 8712 hybrid_clock.cc:648] HybridClock initialized: now 1790577154761434 us; error 41 us; skew 500 ppm
I20260928 06:32:34.762871 8712 webserver.cc:460] Webserver started at http://127.22.225.130:41107/ using document root <none> and password file <none>
I20260928 06:32:34.763134 8712 fs_manager.cc:359] Metadata directory not provided
I20260928 06:32:34.763208 8712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:32:34.763365 8712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:32:34.764691 8712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576899143097-23430-0/minicluster-data/ts-1/data/instance:
uuid: "89744cced02e407aacbdadacdbbac06f"
format_stamp: "Formatted at 2026-09-28 06:32:34 on dist-test-slave-d59m"
server_key: "df3e891ff6f8a7d467ed52fa6c8c157c"
server_key_iv: "afc83ad6e65acfcd2c18c42d082442d4"
server_key_version: "encryptionkey@0"
I20260928 06:32:34.765077 8712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576899143097-23430-0/minicluster-data/ts-1/wal/instance:
uuid: "89744cced02e407aacbdadacdbbac06f"
format_stamp: "Formatted at 2026-09-28 06:32:34 on dist-test-slave-d59m"
server_key: "df3e891ff6f8a7d467ed52fa6c8c157c"
server_key_iv: "afc83ad6e65acfcd2c18c42d082442d4"
server_key_version: "encryptionkey@0"
I20260928 06:32:34.766466 8712 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 06:32:34.767238 8726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:34.767432 8712 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:32:34.767534 8712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576899143097-23430-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576899143097-23430-0/minicluster-data/ts-1/wal
uuid: "89744cced02e407aacbdadacdbbac06f"
format_stamp: "Formatted at 2026-09-28 06:32:34 on dist-test-slave-d59m"
server_key: "df3e891ff6f8a7d467ed52fa6c8c157c"
server_key_iv: "afc83ad6e65acfcd2c18c42d082442d4"
server_key_version: "encryptionkey@0"
I20260928 06:32:34.767627 8712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576899143097-23430-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576899143097-23430-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576899143097-23430-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:32:34.777213 8712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:32:34.777523 8712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:32:34.777757 8712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:32:34.778051 8712 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:32:34.778524 8712 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:32:34.778581 8712 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:34.778662 8712 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:32:34.778697 8712 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:34.785835 8712 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.130:36307
I20260928 06:32:34.785871 8839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.130:36307 every 8 connection(s)
I20260928 06:32:34.786199 8712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
I20260928 06:32:34.792249 8561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:34.786969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:36899 (local address 127.22.225.190:33453)
0928 06:32:34.787043 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:34.787046 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:34.787338 (+ 292us) server_negotiation.cc:407] Connection header received
0928 06:32:34.787983 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:34.787984 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:34.787994 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:34.788038 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:32:34.788886 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:34.789367 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:34.790809 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:34.790998 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:34.791090 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:34.791949 (+ 859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:32:34.791950 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:32:34.791951 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:32:34.791956 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:32:34.791977 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:32:34.791982 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:32:34.792043 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:34.792138 (+ 95us) server_negotiation.cc:299] Negotiation successful
0928 06:32:34.792170 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 06:32:34.792469 8842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:34.786949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:33453 (local address 127.22.225.130:36899)
0928 06:32:34.787188 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:34.787234 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 06:32:34.787868 (+ 634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:32:34.788117 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:32:34.788121 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:32:34.788132 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:32:34.788801 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:34.788807 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:34.789459 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:34.789460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:34.790723 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:34.790725 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:34.791098 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:34.791100 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:34.791241 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:34.791806 (+ 565us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:32:34.791806 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:32:34.791834 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:32:34.791856 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:32:34.792045 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:32:34.792051 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:32:34.792054 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:32:34.792139 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 06:32:34.792247 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":116,"threads_started":1}
I20260928 06:32:34.792995 8840 heartbeater.cc:347] Connected to a master server at 127.22.225.190:33453
I20260928 06:32:34.793098 8840 heartbeater.cc:464] Registering TS with master...
I20260928 06:32:34.793288 8840 heartbeater.cc:511] Master 127.22.225.190:33453 requested a full tablet report, sending...
I20260928 06:32:34.793651 8521 ts_manager.cc:194] Registered new tserver with Master: 89744cced02e407aacbdadacdbbac06f (127.22.225.130:36307)
I20260928 06:32:34.794036 8521 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.225.130:36899
I20260928 06:32:34.794325 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 8712
I20260928 06:32:34.794432 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576899143097-23430-0/minicluster-data/ts-1/wal/instance
I20260928 06:32:34.794667 23430 external_mini_cluster.cc:1428] Setting key f514a335dcd28dfe4dc778d046a63f56
I20260928 06:32:34.795646 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576899143097-23430-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576899143097-23430-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576899143097-23430-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.131:0
--local_ip_for_outbound_sockets=127.22.225.131
--webserver_interface=127.22.225.131
--webserver_port=0
--tserver_master_addrs=127.22.225.190:33453
--builtin_ntp_servers=127.22.225.148:36625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 06:32:34.882671 8843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:32:34.882902 8843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:32:34.882941 8843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:32:34.884279 8843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:32:34.884337 8843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:32:34.884383 8843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.131
I20260928 06:32:34.885964 8843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:36625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576899143097-23430-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576899143097-23430-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.225.131
--webserver_port=0
--tserver_master_addrs=127.22.225.190:33453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.131
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576899143097-23430-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:32:34.886240 8843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:32:34.886543 8843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:32:34.891719 8849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:34.891805 8851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:34.891726 8848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:32:34.892182 8843 server_base.cc:1034] running on GCE node
I20260928 06:32:34.892396 8843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:32:34.892709 8843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:32:34.893880 8843 hybrid_clock.cc:648] HybridClock initialized: now 1790577154893861 us; error 35 us; skew 500 ppm
I20260928 06:32:34.895141 8843 webserver.cc:460] Webserver started at http://127.22.225.131:42143/ using document root <none> and password file <none>
I20260928 06:32:34.895435 8843 fs_manager.cc:359] Metadata directory not provided
I20260928 06:32:34.895505 8843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:32:34.895648 8843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:32:34.896821 8843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576899143097-23430-0/minicluster-data/ts-2/data/instance:
uuid: "b7b47754b1794357aada0562bc47e574"
format_stamp: "Formatted at 2026-09-28 06:32:34 on dist-test-slave-d59m"
server_key: "6de66a371211270872b79333db985c46"
server_key_iv: "b0923a5df15959673860158217490272"
server_key_version: "encryptionkey@0"
I20260928 06:32:34.897202 8843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576899143097-23430-0/minicluster-data/ts-2/wal/instance:
uuid: "b7b47754b1794357aada0562bc47e574"
format_stamp: "Formatted at 2026-09-28 06:32:34 on dist-test-slave-d59m"
server_key: "6de66a371211270872b79333db985c46"
server_key_iv: "b0923a5df15959673860158217490272"
server_key_version: "encryptionkey@0"
I20260928 06:32:34.898568 8843 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:32:34.899367 8857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:34.899533 8843 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:32:34.899655 8843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576899143097-23430-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576899143097-23430-0/minicluster-data/ts-2/wal
uuid: "b7b47754b1794357aada0562bc47e574"
format_stamp: "Formatted at 2026-09-28 06:32:34 on dist-test-slave-d59m"
server_key: "6de66a371211270872b79333db985c46"
server_key_iv: "b0923a5df15959673860158217490272"
server_key_version: "encryptionkey@0"
I20260928 06:32:34.899768 8843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576899143097-23430-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576899143097-23430-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576899143097-23430-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:32:34.912482 8843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:32:34.912949 8843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:32:34.913116 8843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:32:34.913368 8843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:32:34.913728 8843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:32:34.913784 8843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:34.913829 8843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:32:34.913856 8843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:34.920802 8843 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.131:35609
I20260928 06:32:34.920835 8970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.131:35609 every 8 connection(s)
I20260928 06:32:34.921195 8843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
I20260928 06:32:34.926842 8561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:34.921990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:48363 (local address 127.22.225.190:33453)
0928 06:32:34.922063 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:34.922065 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:34.922313 (+ 248us) server_negotiation.cc:407] Connection header received
0928 06:32:34.922822 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:34.922824 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:34.922836 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:34.922892 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:32:34.923630 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:34.924141 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:34.925285 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:34.925495 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:34.925579 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:34.926501 (+ 922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:32:34.926502 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:32:34.926502 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:32:34.926508 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:32:34.926531 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:32:34.926535 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:32:34.926598 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:34.926715 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 06:32:34.926749 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 06:32:34.927119 8973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:34.921957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:33453 (local address 127.22.225.131:48363)
0928 06:32:34.922160 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:34.922213 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 06:32:34.922708 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:32:34.922984 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:32:34.922989 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:32:34.922997 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:32:34.923526 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:34.923532 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:34.924279 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:34.924280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:34.925176 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:34.925180 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:34.925587 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:34.925588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:34.925731 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:34.926338 (+ 607us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:32:34.926338 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:32:34.926369 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:32:34.926395 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:32:34.926598 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:32:34.926605 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:32:34.926609 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:32:34.926697 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 06:32:34.926815 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":89,"threads_started":1}
I20260928 06:32:34.927677 8971 heartbeater.cc:347] Connected to a master server at 127.22.225.190:33453
I20260928 06:32:34.927776 8971 heartbeater.cc:464] Registering TS with master...
I20260928 06:32:34.927937 8971 heartbeater.cc:511] Master 127.22.225.190:33453 requested a full tablet report, sending...
I20260928 06:32:34.928323 8521 ts_manager.cc:194] Registered new tserver with Master: b7b47754b1794357aada0562bc47e574 (127.22.225.131:35609)
I20260928 06:32:34.928736 8521 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.225.131:48363
I20260928 06:32:34.930842 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 8843
I20260928 06:32:34.930951 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576899143097-23430-0/minicluster-data/ts-2/wal/instance
I20260928 06:32:34.931166 23430 external_mini_cluster.cc:1428] Setting key 47cc401d383b0d22589db919f1b2766c
I20260928 06:32:34.932113 23430 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:32:34.937101 8561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:34.933730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57890 (local address 127.22.225.190:33453)
0928 06:32:34.933788 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:34.933790 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:34.933927 (+ 137us) server_negotiation.cc:407] Connection header received
0928 06:32:34.933987 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:34.933988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:34.934000 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:34.934048 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:32:34.934225 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:34.934685 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:34.935825 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:34.936009 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:34.936068 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:34.936847 (+ 779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:32:34.936848 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:32:34.936849 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:32:34.936853 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:32:34.936866 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:32:34.936870 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:32:34.936915 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:34.936982 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 06:32:34.937003 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 06:32:34.938813 8521 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57890:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:32:34.940378 8521 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:32:34.948145 8983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:34.942924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.129:40789 (local address 127.0.0.1:41716)
0928 06:32:34.943446 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:34.943491 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 06:32:34.943593 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:32:34.943906 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:32:34.943910 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:32:34.943912 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:32:34.943992 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:34.943995 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:34.945408 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:34.945409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:34.946755 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:34.946758 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:34.947838 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:34.947839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:34.947916 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:34.947928 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 06:32:34.947988 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 06:32:34.948046 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":75,"threads_started":1}
I20260928 06:32:34.948607 8980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:34.943025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41716 (local address 127.22.225.129:40789)
0928 06:32:34.943340 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:34.943346 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:34.943565 (+ 219us) server_negotiation.cc:407] Connection header received
0928 06:32:34.943718 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:34.943721 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:34.943740 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:34.943815 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:32:34.944083 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:34.945300 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:34.946862 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:34.947641 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:34.947693 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:34.947778 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:34.948453 (+ 675us) server_negotiation.cc:299] Negotiation successful
0928 06:32:34.948495 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":139,"threads_started":1}
I20260928 06:32:34.948696 8982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:34.943282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55938 (local address 127.22.225.130:36307)
0928 06:32:34.943757 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:34.943765 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:34.943773 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:32:34.943809 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:34.943810 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:34.943824 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:34.943891 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:32:34.944072 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:34.945299 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:34.947158 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:34.947892 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:34.948225 (+ 333us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:34.948329 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:34.948532 (+ 203us) server_negotiation.cc:299] Negotiation successful
0928 06:32:34.948601 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":102,"threads_started":1}
I20260928 06:32:34.949291 8979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:34.943308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.131:35609 (local address 127.0.0.1:53258)
0928 06:32:34.943942 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:34.943951 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:32:34.943991 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:32:34.945699 (+ 1708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:32:34.945700 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:32:34.945702 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:32:34.945738 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:34.945739 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:34.946954 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:34.946955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:34.947997 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:34.947999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:34.949059 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:34.949060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:34.949128 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:34.949131 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:32:34.949182 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 06:32:34.949224 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":80,"threads_started":1}
I20260928 06:32:34.949771 8643 tablet_service.cc:1467] Processing CreateTablet for tablet 2ded842bbeb74fc2b62ab711e570e4d5 (DEFAULT_TABLE table=test-table [id=9b6fe090b56a4483a372570b1a85c390]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:32:34.949911 8984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:34.943513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53258 (local address 127.22.225.131:35609)
0928 06:32:34.945472 (+ 1959us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:34.945476 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:34.945485 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:32:34.945522 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:34.945525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:34.945540 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:34.945615 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:32:34.945815 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:34.946871 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:34.948081 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:34.948867 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:34.948938 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:34.949022 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:34.949782 (+ 760us) server_negotiation.cc:299] Negotiation successful
0928 06:32:34.949826 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1914,"thread_start_us":81,"threads_started":1}
I20260928 06:32:34.948895 8981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:34.943184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.130:36307 (local address 127.0.0.1:55938)
0928 06:32:34.943458 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:34.943498 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 06:32:34.943566 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:32:34.943953 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:32:34.943954 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:32:34.943955 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:32:34.943994 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:34.943994 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:34.945391 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:34.945393 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:34.947049 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:34.947053 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:34.948388 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:34.948388 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:34.948454 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:34.948457 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:32:34.948776 (+ 319us) client_negotiation.cc:240] Negotiation successful
0928 06:32:34.948819 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":56,"threads_started":1}
I20260928 06:32:34.950209 8643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ded842bbeb74fc2b62ab711e570e4d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:32:34.950819 8774 tablet_service.cc:1467] Processing CreateTablet for tablet 2ded842bbeb74fc2b62ab711e570e4d5 (DEFAULT_TABLE table=test-table [id=9b6fe090b56a4483a372570b1a85c390]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:32:34.950898 8905 tablet_service.cc:1467] Processing CreateTablet for tablet 2ded842bbeb74fc2b62ab711e570e4d5 (DEFAULT_TABLE table=test-table [id=9b6fe090b56a4483a372570b1a85c390]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:32:34.951171 8774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ded842bbeb74fc2b62ab711e570e4d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:32:34.951221 8905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ded842bbeb74fc2b62ab711e570e4d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:32:34.954196 8985 tablet_bootstrap.cc:492] T 2ded842bbeb74fc2b62ab711e570e4d5 P c5ecb1a8717342febf88ac86b45ebe2f: Bootstrap starting.
I20260928 06:32:34.954818 8987 tablet_bootstrap.cc:492] T 2ded842bbeb74fc2b62ab711e570e4d5 P 89744cced02e407aacbdadacdbbac06f: Bootstrap starting.
I20260928 06:32:34.954967 8985 tablet_bootstrap.cc:654] T 2ded842bbeb74fc2b62ab711e570e4d5 P c5ecb1a8717342febf88ac86b45ebe2f: Neither blocks nor log segments found. Creating new log.
I20260928 06:32:34.955320 8985 log.cc:824] T 2ded842bbeb74fc2b62ab711e570e4d5 P c5ecb1a8717342febf88ac86b45ebe2f: Log is configured to *not* fsync() on all Append() calls
I20260928 06:32:34.955708 8987 tablet_bootstrap.cc:654] T 2ded842bbeb74fc2b62ab711e570e4d5 P 89744cced02e407aacbdadacdbbac06f: Neither blocks nor log segments found. Creating new log.
I20260928 06:32:34.955693 8986 tablet_bootstrap.cc:492] T 2ded842bbeb74fc2b62ab711e570e4d5 P b7b47754b1794357aada0562bc47e574: Bootstrap starting.
I20260928 06:32:34.956176 8987 log.cc:824] T 2ded842bbeb74fc2b62ab711e570e4d5 P 89744cced02e407aacbdadacdbbac06f: Log is configured to *not* fsync() on all Append() calls
I20260928 06:32:34.956338 8986 tablet_bootstrap.cc:654] T 2ded842bbeb74fc2b62ab711e570e4d5 P b7b47754b1794357aada0562bc47e574: Neither blocks nor log segments found. Creating new log.
I20260928 06:32:34.956707 8986 log.cc:824] T 2ded842bbeb74fc2b62ab711e570e4d5 P b7b47754b1794357aada0562bc47e574: Log is configured to *not* fsync() on all Append() calls
I20260928 06:32:34.957206 8987 tablet_bootstrap.cc:492] T 2ded842bbeb74fc2b62ab711e570e4d5 P 89744cced02e407aacbdadacdbbac06f: No bootstrap required, opened a new log
I20260928 06:32:34.957242 8985 tablet_bootstrap.cc:492] T 2ded842bbeb74fc2b62ab711e570e4d5 P c5ecb1a8717342febf88ac86b45ebe2f: No bootstrap required, opened a new log
I20260928 06:32:34.957309 8985 ts_tablet_manager.cc:1397] T 2ded842bbeb74fc2b62ab711e570e4d5 P c5ecb1a8717342febf88ac86b45ebe2f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 06:32:34.957338 8987 ts_tablet_manager.cc:1397] T 2ded842bbeb74fc2b62ab711e570e4d5 P 89744cced02e407aacbdadacdbbac06f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 06:32:34.960044 8985 raft_consensus.cc:348] T 2ded842bbeb74fc2b62ab711e570e4d5 P c5ecb1a8717342febf88ac86b45ebe2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ecb1a8717342febf88ac86b45ebe2f" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 40789 } } peers { permanent_uuid: "89744cced02e407aacbdadacdbbac06f" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 36307 } } peers { permanent_uuid: "b7b47754b1794357aada0562bc47e574" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 35609 } }
I20260928 06:32:34.960247 8985 raft_consensus.cc:374] T 2ded842bbeb74fc2b62ab711e570e4d5 P c5ecb1a8717342febf88ac86b45ebe2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:32:34.960291 8985 raft_consensus.cc:729] T 2ded842bbeb74fc2b62ab711e570e4d5 P c5ecb1a8717342febf88ac86b45ebe2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ecb1a8717342febf88ac86b45ebe2f, State: Initialized, Role: FOLLOWER
I20260928 06:32:34.960148 8987 raft_consensus.cc:348] T 2ded842bbeb74fc2b62ab711e570e4d5 P 89744cced02e407aacbdadacdbbac06f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ecb1a8717342febf88ac86b45ebe2f" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 40789 } } peers { permanent_uuid: "89744cced02e407aacbdadacdbbac06f" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 36307 } } peers { permanent_uuid: "b7b47754b1794357aada0562bc47e574" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 35609 } }
I20260928 06:32:34.960423 8985 consensus_queue.cc:260] T 2ded842bbeb74fc2b62ab711e570e4d5 P c5ecb1a8717342febf88ac86b45ebe2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ecb1a8717342febf88ac86b45ebe2f" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 40789 } } peers { permanent_uuid: "89744cced02e407aacbdadacdbbac06f" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 36307 } } peers { permanent_uuid: "b7b47754b1794357aada0562bc47e574" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 35609 } }
I20260928 06:32:34.960608 8987 raft_consensus.cc:374] T 2ded842bbeb74fc2b62ab711e570e4d5 P 89744cced02e407aacbdadacdbbac06f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:32:34.960703 8987 raft_consensus.cc:729] T 2ded842bbeb74fc2b62ab711e570e4d5 P 89744cced02e407aacbdadacdbbac06f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89744cced02e407aacbdadacdbbac06f, State: Initialized, Role: FOLLOWER
I20260928 06:32:34.960894 8987 consensus_queue.cc:260] T 2ded842bbeb74fc2b62ab711e570e4d5 P 89744cced02e407aacbdadacdbbac06f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ecb1a8717342febf88ac86b45ebe2f" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 40789 } } peers { permanent_uuid: "89744cced02e407aacbdadacdbbac06f" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 36307 } } peers { permanent_uuid: "b7b47754b1794357aada0562bc47e574" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 35609 } }
I20260928 06:32:34.961208 8986 tablet_bootstrap.cc:492] T 2ded842bbeb74fc2b62ab711e570e4d5 P b7b47754b1794357aada0562bc47e574: No bootstrap required, opened a new log
I20260928 06:32:34.961294 8986 ts_tablet_manager.cc:1397] T 2ded842bbeb74fc2b62ab711e570e4d5 P b7b47754b1794357aada0562bc47e574: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260928 06:32:34.961297 8840 heartbeater.cc:503] Master 127.22.225.190:33453 was elected leader, sending a full tablet report...
I20260928 06:32:34.961241 8987 ts_tablet_manager.cc:1428] T 2ded842bbeb74fc2b62ab711e570e4d5 P 89744cced02e407aacbdadacdbbac06f: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260928 06:32:34.963758 8986 raft_consensus.cc:348] T 2ded842bbeb74fc2b62ab711e570e4d5 P b7b47754b1794357aada0562bc47e574 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ecb1a8717342febf88ac86b45ebe2f" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 40789 } } peers { permanent_uuid: "89744cced02e407aacbdadacdbbac06f" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 36307 } } peers { permanent_uuid: "b7b47754b1794357aada0562bc47e574" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 35609 } }
I20260928 06:32:34.963940 8986 raft_consensus.cc:374] T 2ded842bbeb74fc2b62ab711e570e4d5 P b7b47754b1794357aada0562bc47e574 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:32:34.963981 8986 raft_consensus.cc:729] T 2ded842bbeb74fc2b62ab711e570e4d5 P b7b47754b1794357aada0562bc47e574 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7b47754b1794357aada0562bc47e574, State: Initialized, Role: FOLLOWER
I20260928 06:32:34.964103 8986 consensus_queue.cc:260] T 2ded842bbeb74fc2b62ab711e570e4d5 P b7b47754b1794357aada0562bc47e574 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ecb1a8717342febf88ac86b45ebe2f" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 40789 } } peers { permanent_uuid: "89744cced02e407aacbdadacdbbac06f" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 36307 } } peers { permanent_uuid: "b7b47754b1794357aada0562bc47e574" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 35609 } }
I20260928 06:32:34.964388 8986 ts_tablet_manager.cc:1428] T 2ded842bbeb74fc2b62ab711e570e4d5 P b7b47754b1794357aada0562bc47e574: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 06:32:34.964404 8971 heartbeater.cc:503] Master 127.22.225.190:33453 was elected leader, sending a full tablet report...
I20260928 06:32:34.960795 8709 heartbeater.cc:503] Master 127.22.225.190:33453 was elected leader, sending a full tablet report...
I20260928 06:32:34.960757 8985 ts_tablet_manager.cc:1428] T 2ded842bbeb74fc2b62ab711e570e4d5 P c5ecb1a8717342febf88ac86b45ebe2f: Time spent starting tablet: real 0.003s user 0.001s sys 0.003s
W20260928 06:32:35.036811 8841 tablet.cc:2369] T 2ded842bbeb74fc2b62ab711e570e4d5 P 89744cced02e407aacbdadacdbbac06f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:32:35.037093 8992 raft_consensus.cc:482] T 2ded842bbeb74fc2b62ab711e570e4d5 P 89744cced02e407aacbdadacdbbac06f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:32:35.037248 8992 raft_consensus.cc:504] T 2ded842bbeb74fc2b62ab711e570e4d5 P 89744cced02e407aacbdadacdbbac06f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ecb1a8717342febf88ac86b45ebe2f" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 40789 } } peers { permanent_uuid: "89744cced02e407aacbdadacdbbac06f" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 36307 } } peers { permanent_uuid: "b7b47754b1794357aada0562bc47e574" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 35609 } }
I20260928 06:32:35.037629 8992 leader_election.cc:288] T 2ded842bbeb74fc2b62ab711e570e4d5 P 89744cced02e407aacbdadacdbbac06f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5ecb1a8717342febf88ac86b45ebe2f (127.22.225.129:40789), b7b47754b1794357aada0562bc47e574 (127.22.225.131:35609)
I20260928 06:32:35.041286 8842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:35.037833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.131:35609 (local address 127.22.225.130:51803)
0928 06:32:35.037903 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:35.037916 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:32:35.037961 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:32:35.038284 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:32:35.038285 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:32:35.038287 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:32:35.038361 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:35.038362 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:35.039118 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:35.039119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:35.040356 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:35.040358 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:35.040993 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:35.040994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:35.041067 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:35.041070 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:32:35.041126 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 06:32:35.041176 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":4}
I20260928 06:32:35.041402 8980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:35.038005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:39285 (local address 127.22.225.129:40789)
0928 06:32:35.038084 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:35.038087 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:35.038115 (+ 28us) server_negotiation.cc:407] Connection header received
0928 06:32:35.038175 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:35.038176 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:35.038187 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:35.038255 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:32:35.038431 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:35.039187 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:35.040462 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:35.041046 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:35.041121 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:35.041129 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:35.041278 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 06:32:35.041317 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 06:32:35.041497 8994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:35.037847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.129:40789 (local address 127.22.225.130:39285)
0928 06:32:35.038053 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:35.038063 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:32:35.038103 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:32:35.038262 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:32:35.038263 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:32:35.038264 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:32:35.038338 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:35.038339 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:35.039273 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:35.039273 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:35.040357 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:35.040359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:35.041132 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:35.041133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:35.041204 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:35.041207 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:32:35.041261 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 06:32:35.041291 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":1,"thread_start_us":116,"threads_started":1}
I20260928 06:32:35.041906 8984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:35.038058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:51803 (local address 127.22.225.131:35609)
0928 06:32:35.038106 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:35.038108 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:35.038115 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:32:35.038148 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:35.038148 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:35.038159 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:35.038236 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:32:35.038446 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:35.039023 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:35.040480 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:35.040907 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:35.040993 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:35.041002 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:35.041798 (+ 796us) server_negotiation.cc:299] Negotiation successful
0928 06:32:35.041843 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 06:32:35.041898 8663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ded842bbeb74fc2b62ab711e570e4d5" candidate_uuid: "89744cced02e407aacbdadacdbbac06f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ecb1a8717342febf88ac86b45ebe2f" is_pre_election: true
I20260928 06:32:35.042063 8663 raft_consensus.cc:2454] T 2ded842bbeb74fc2b62ab711e570e4d5 P c5ecb1a8717342febf88ac86b45ebe2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89744cced02e407aacbdadacdbbac06f in term 0.
I20260928 06:32:35.042176 8925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ded842bbeb74fc2b62ab711e570e4d5" candidate_uuid: "89744cced02e407aacbdadacdbbac06f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7b47754b1794357aada0562bc47e574" is_pre_election: true
I20260928 06:32:35.042301 8729 leader_election.cc:302] T 2ded842bbeb74fc2b62ab711e570e4d5 P 89744cced02e407aacbdadacdbbac06f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89744cced02e407aacbdadacdbbac06f, c5ecb1a8717342febf88ac86b45ebe2f; no voters:
I20260928 06:32:35.042326 8925 raft_consensus.cc:2454] T 2ded842bbeb74fc2b62ab711e570e4d5 P b7b47754b1794357aada0562bc47e574 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89744cced02e407aacbdadacdbbac06f in term 0.
I20260928 06:32:35.042451 8992 raft_consensus.cc:2781] T 2ded842bbeb74fc2b62ab711e570e4d5 P 89744cced02e407aacbdadacdbbac06f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:32:35.042517 8992 raft_consensus.cc:482] T 2ded842bbeb74fc2b62ab711e570e4d5 P 89744cced02e407aacbdadacdbbac06f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:32:35.042551 8992 raft_consensus.cc:3037] T 2ded842bbeb74fc2b62ab711e570e4d5 P 89744cced02e407aacbdadacdbbac06f [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:32:35.043246 8992 raft_consensus.cc:504] T 2ded842bbeb74fc2b62ab711e570e4d5 P 89744cced02e407aacbdadacdbbac06f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ecb1a8717342febf88ac86b45ebe2f" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 40789 } } peers { permanent_uuid: "89744cced02e407aacbdadacdbbac06f" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 36307 } } peers { permanent_uuid: "b7b47754b1794357aada0562bc47e574" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 35609 } }
I20260928 06:32:35.043388 8992 leader_election.cc:288] T 2ded842bbeb74fc2b62ab711e570e4d5 P 89744cced02e407aacbdadacdbbac06f [CANDIDATE]: Term 1 election: Requested vote from peers c5ecb1a8717342febf88ac86b45ebe2f (127.22.225.129:40789), b7b47754b1794357aada0562bc47e574 (127.22.225.131:35609)
I20260928 06:32:35.043572 8925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ded842bbeb74fc2b62ab711e570e4d5" candidate_uuid: "89744cced02e407aacbdadacdbbac06f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7b47754b1794357aada0562bc47e574"
I20260928 06:32:35.043674 8925 raft_consensus.cc:3037] T 2ded842bbeb74fc2b62ab711e570e4d5 P b7b47754b1794357aada0562bc47e574 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:32:35.043823 8663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ded842bbeb74fc2b62ab711e570e4d5" candidate_uuid: "89744cced02e407aacbdadacdbbac06f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ecb1a8717342febf88ac86b45ebe2f"
I20260928 06:32:35.043907 8663 raft_consensus.cc:3037] T 2ded842bbeb74fc2b62ab711e570e4d5 P c5ecb1a8717342febf88ac86b45ebe2f [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:32:35.044656 8925 raft_consensus.cc:2454] T 2ded842bbeb74fc2b62ab711e570e4d5 P b7b47754b1794357aada0562bc47e574 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89744cced02e407aacbdadacdbbac06f in term 1.
I20260928 06:32:35.044817 8730 leader_election.cc:302] T 2ded842bbeb74fc2b62ab711e570e4d5 P 89744cced02e407aacbdadacdbbac06f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89744cced02e407aacbdadacdbbac06f, b7b47754b1794357aada0562bc47e574; no voters:
I20260928 06:32:35.044891 8663 raft_consensus.cc:2454] T 2ded842bbeb74fc2b62ab711e570e4d5 P c5ecb1a8717342febf88ac86b45ebe2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89744cced02e407aacbdadacdbbac06f in term 1.
I20260928 06:32:35.044910 8992 raft_consensus.cc:2781] T 2ded842bbeb74fc2b62ab711e570e4d5 P 89744cced02e407aacbdadacdbbac06f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:32:35.045058 8992 raft_consensus.cc:686] T 2ded842bbeb74fc2b62ab711e570e4d5 P 89744cced02e407aacbdadacdbbac06f [term 1 LEADER]: Becoming Leader. State: Replica: 89744cced02e407aacbdadacdbbac06f, State: Running, Role: LEADER
I20260928 06:32:35.045174 8992 consensus_queue.cc:237] T 2ded842bbeb74fc2b62ab711e570e4d5 P 89744cced02e407aacbdadacdbbac06f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ecb1a8717342febf88ac86b45ebe2f" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 40789 } } peers { permanent_uuid: "89744cced02e407aacbdadacdbbac06f" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 36307 } } peers { permanent_uuid: "b7b47754b1794357aada0562bc47e574" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 35609 } }
I20260928 06:32:35.046020 8521 catalog_manager.cc:5495] T 2ded842bbeb74fc2b62ab711e570e4d5 P 89744cced02e407aacbdadacdbbac06f reported cstate change: term changed from 0 to 1, leader changed from <none> to 89744cced02e407aacbdadacdbbac06f (127.22.225.130). New cstate: current_term: 1 leader_uuid: "89744cced02e407aacbdadacdbbac06f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ecb1a8717342febf88ac86b45ebe2f" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 40789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89744cced02e407aacbdadacdbbac06f" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 36307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b7b47754b1794357aada0562bc47e574" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 35609 } health_report { overall_health: UNKNOWN } } }
I20260928 06:32:35.066692 8982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:35.063979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55944 (local address 127.22.225.130:36307)
0928 06:32:35.064033 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:35.064036 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:35.064047 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:32:35.064103 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:35.064104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:35.064128 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:35.064222 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:32:35.064404 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:35.064994 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:35.065798 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:35.066012 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:35.066106 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:35.066237 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:32:35.066360 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:32:35.066488 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:35.066548 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 06:32:35.066568 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 06:32:35.068609 8925 raft_consensus.cc:1261] T 2ded842bbeb74fc2b62ab711e570e4d5 P b7b47754b1794357aada0562bc47e574 [term 1 FOLLOWER]: Refusing update from remote peer 89744cced02e407aacbdadacdbbac06f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:32:35.068634 8663 raft_consensus.cc:1261] T 2ded842bbeb74fc2b62ab711e570e4d5 P c5ecb1a8717342febf88ac86b45ebe2f [term 1 FOLLOWER]: Refusing update from remote peer 89744cced02e407aacbdadacdbbac06f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:32:35.068965 8995 consensus_queue.cc:1035] T 2ded842bbeb74fc2b62ab711e570e4d5 P 89744cced02e407aacbdadacdbbac06f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7b47754b1794357aada0562bc47e574" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 35609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:32:35.069108 8992 consensus_queue.cc:1035] T 2ded842bbeb74fc2b62ab711e570e4d5 P 89744cced02e407aacbdadacdbbac06f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ecb1a8717342febf88ac86b45ebe2f" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 40789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:32:35.072052 8998 mvcc.cc:205] Tried to move back new op lower bound from 7334204027158138880 to 7334204027065737216. Current Snapshot: MvccSnapshot[applied={T|T < 7334204027158138880}]
I20260928 06:32:35.072957 8999 mvcc.cc:205] Tried to move back new op lower bound from 7334204027158138880 to 7334204027065737216. Current Snapshot: MvccSnapshot[applied={T|T < 7334204027158138880}]
I20260928 06:32:35.074151 8520 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57890:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:32:35.074252 8520 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57890:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:32:35.075335 8520 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9bed558d95154b8daa90c0e990412ceb: Sending DeleteTablet for 3 replicas of tablet 2ded842bbeb74fc2b62ab711e570e4d5
I20260928 06:32:35.075865 8774 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ded842bbeb74fc2b62ab711e570e4d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:32:35 UTC) from {username='slave'} at 127.0.0.1:55938
I20260928 06:32:35.075935 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 8581
I20260928 06:32:35.076088 8905 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ded842bbeb74fc2b62ab711e570e4d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:32:35 UTC) from {username='slave'} at 127.0.0.1:53258
I20260928 06:32:35.076654 9008 tablet_replica.cc:330] T 2ded842bbeb74fc2b62ab711e570e4d5 P b7b47754b1794357aada0562bc47e574: stopping tablet replica
I20260928 06:32:35.076795 9008 raft_consensus.cc:2229] T 2ded842bbeb74fc2b62ab711e570e4d5 P b7b47754b1794357aada0562bc47e574 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:32:35.076952 9008 raft_consensus.cc:2258] T 2ded842bbeb74fc2b62ab711e570e4d5 P b7b47754b1794357aada0562bc47e574 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:32:35.077473 9008 ts_tablet_manager.cc:1905] T 2ded842bbeb74fc2b62ab711e570e4d5 P b7b47754b1794357aada0562bc47e574: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:32:35.078806 9007 tablet_replica.cc:330] T 2ded842bbeb74fc2b62ab711e570e4d5 P 89744cced02e407aacbdadacdbbac06f: stopping tablet replica
I20260928 06:32:35.078928 9007 raft_consensus.cc:2229] T 2ded842bbeb74fc2b62ab711e570e4d5 P 89744cced02e407aacbdadacdbbac06f [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:32:35.079082 9007 raft_consensus.cc:2258] T 2ded842bbeb74fc2b62ab711e570e4d5 P 89744cced02e407aacbdadacdbbac06f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:32:35.079093 9008 ts_tablet_manager.cc:1918] T 2ded842bbeb74fc2b62ab711e570e4d5 P b7b47754b1794357aada0562bc47e574: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:32:35.079192 9008 log.cc:1196] T 2ded842bbeb74fc2b62ab711e570e4d5 P b7b47754b1794357aada0562bc47e574: Deleting WAL directory at /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576899143097-23430-0/minicluster-data/ts-2/wal/wals/2ded842bbeb74fc2b62ab711e570e4d5
I20260928 06:32:35.079715 9008 ts_tablet_manager.cc:1939] T 2ded842bbeb74fc2b62ab711e570e4d5 P b7b47754b1794357aada0562bc47e574: Deleting consensus metadata
I20260928 06:32:35.080219 8508 catalog_manager.cc:4841] TS b7b47754b1794357aada0562bc47e574 (127.22.225.131:35609): tablet 2ded842bbeb74fc2b62ab711e570e4d5 (table test-table [id=9b6fe090b56a4483a372570b1a85c390]) successfully deleted
I20260928 06:32:35.081410 9007 ts_tablet_manager.cc:1905] T 2ded842bbeb74fc2b62ab711e570e4d5 P 89744cced02e407aacbdadacdbbac06f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:32:35.082726 9007 ts_tablet_manager.cc:1918] T 2ded842bbeb74fc2b62ab711e570e4d5 P 89744cced02e407aacbdadacdbbac06f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:32:35.082852 9007 log.cc:1196] T 2ded842bbeb74fc2b62ab711e570e4d5 P 89744cced02e407aacbdadacdbbac06f: Deleting WAL directory at /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576899143097-23430-0/minicluster-data/ts-1/wal/wals/2ded842bbeb74fc2b62ab711e570e4d5
I20260928 06:32:35.083142 9007 ts_tablet_manager.cc:1939] T 2ded842bbeb74fc2b62ab711e570e4d5 P 89744cced02e407aacbdadacdbbac06f: Deleting consensus metadata
I20260928 06:32:35.084734 8508 catalog_manager.cc:4841] TS 89744cced02e407aacbdadacdbbac06f (127.22.225.130:36307): tablet 2ded842bbeb74fc2b62ab711e570e4d5 (table test-table [id=9b6fe090b56a4483a372570b1a85c390]) successfully deleted
I20260928 06:32:35.086218 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 8712
W20260928 06:32:35.086583 8507 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.225.129:40789 (error 108)
I20260928 06:32:35.088821 8981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:35.088661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.129:40789 (local address 127.0.0.1:41724)
0928 06:32:35.088720 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:35.088783 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.129:40789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 06:32:35.088949 8507 catalog_manager.cc:4584] TS c5ecb1a8717342febf88ac86b45ebe2f (127.22.225.129:40789): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2ded842bbeb74fc2b62ab711e570e4d5: Network error: Client connection negotiation failed: client connection to 127.22.225.129:40789: connect: Connection refused (error 111)
I20260928 06:32:35.091120 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 8843
I20260928 06:32:35.096684 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 8490
2026-09-28T06:32:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (850 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:32:35 dist-test-slave-d59m krb5kdc[9012](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:32:35 dist-test-slave-d59m krb5kdc[9012](info): set up 2 sockets
Sep 28 06:32:35 dist-test-slave-d59m krb5kdc[9012](info): commencing operation
krb5kdc: starting...
W20260928 06:32:36.177557 23430 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Sep 28 06:32:36 dist-test-slave-d59m krb5kdc[9012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:32:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:32:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab.
Entry for principal kudu/127.22.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab.
I20260928 06:32:36.319340 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:38821
--webserver_interface=127.22.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab
--principal=kudu/127.22.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:40447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.225.190:38821
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:32:36.408968 9025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:32:36.409255 9025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:32:36.409338 9025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:32:36.411113 9025 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:32:36.411217 9025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:32:36.411273 9025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:32:36.411324 9025 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:32:36.411378 9025 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:32:36.413829 9025 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:40447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.225.190:38821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab
--principal=kudu/127.22.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:38821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:32:36.414105 9025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:32:36.414356 9025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:32:36.419166 9033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:36.419277 9030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:36.419173 9031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:32:36.419188 9025 server_base.cc:1034] running on GCE node
I20260928 06:32:36.419912 9025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:32:36.420238 9025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:32:36.421435 9025 hybrid_clock.cc:648] HybridClock initialized: now 1790577156421425 us; error 41 us; skew 500 ppm
Sep 28 06:32:36 dist-test-slave-d59m krb5kdc[9012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577156, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:32:36.424537 9025 init.cc:377] Logged in from keytab as kudu/127.22.225.190@KRBTEST.COM (short username kudu)
I20260928 06:32:36.425071 9025 webserver.cc:460] Webserver started at http://127.22.225.190:38071/ using document root <none> and password file <none>
I20260928 06:32:36.425314 9025 fs_manager.cc:359] Metadata directory not provided
I20260928 06:32:36.425385 9025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:32:36.425521 9025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:32:36.426762 9025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/minicluster-data/master-0/data/instance:
uuid: "e598493c02424281bc5e6fada64030e0"
format_stamp: "Formatted at 2026-09-28 06:32:36 on dist-test-slave-d59m"
server_key: "bb95fc24507e5d7ae1e92858dff86aec"
server_key_iv: "2861410da5d0f856b1515a47e849422f"
server_key_version: "encryptionkey@0"
I20260928 06:32:36.427202 9025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/minicluster-data/master-0/wal/instance:
uuid: "e598493c02424281bc5e6fada64030e0"
format_stamp: "Formatted at 2026-09-28 06:32:36 on dist-test-slave-d59m"
server_key: "bb95fc24507e5d7ae1e92858dff86aec"
server_key_iv: "2861410da5d0f856b1515a47e849422f"
server_key_version: "encryptionkey@0"
I20260928 06:32:36.428774 9025 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:32:36.429554 9040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:36.429764 9025 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:32:36.429867 9025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/minicluster-data/master-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/minicluster-data/master-0/wal
uuid: "e598493c02424281bc5e6fada64030e0"
format_stamp: "Formatted at 2026-09-28 06:32:36 on dist-test-slave-d59m"
server_key: "bb95fc24507e5d7ae1e92858dff86aec"
server_key_iv: "2861410da5d0f856b1515a47e849422f"
server_key_version: "encryptionkey@0"
I20260928 06:32:36.429960 9025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:32:36.447119 9025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:32:36.449653 9025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:32:36.449841 9025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:32:36.454442 9025 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.190:38821
I20260928 06:32:36.454486 9092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.190:38821 every 8 connection(s)
I20260928 06:32:36.454871 9025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
I20260928 06:32:36.455910 9093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:32:36.459038 9093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e598493c02424281bc5e6fada64030e0: Bootstrap starting.
I20260928 06:32:36.459892 9093 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e598493c02424281bc5e6fada64030e0: Neither blocks nor log segments found. Creating new log.
I20260928 06:32:36.460255 9093 log.cc:824] T 00000000000000000000000000000000 P e598493c02424281bc5e6fada64030e0: Log is configured to *not* fsync() on all Append() calls
I20260928 06:32:36.461117 9093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e598493c02424281bc5e6fada64030e0: No bootstrap required, opened a new log
I20260928 06:32:36.463347 9093 raft_consensus.cc:348] T 00000000000000000000000000000000 P e598493c02424281bc5e6fada64030e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e598493c02424281bc5e6fada64030e0" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 38821 } }
I20260928 06:32:36.463536 9093 raft_consensus.cc:374] T 00000000000000000000000000000000 P e598493c02424281bc5e6fada64030e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:32:36.463567 9093 raft_consensus.cc:729] T 00000000000000000000000000000000 P e598493c02424281bc5e6fada64030e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e598493c02424281bc5e6fada64030e0, State: Initialized, Role: FOLLOWER
I20260928 06:32:36.463721 9093 consensus_queue.cc:260] T 00000000000000000000000000000000 P e598493c02424281bc5e6fada64030e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e598493c02424281bc5e6fada64030e0" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 38821 } }
I20260928 06:32:36.463793 9093 raft_consensus.cc:388] T 00000000000000000000000000000000 P e598493c02424281bc5e6fada64030e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:32:36.463821 9093 raft_consensus.cc:482] T 00000000000000000000000000000000 P e598493c02424281bc5e6fada64030e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:32:36.463868 9093 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e598493c02424281bc5e6fada64030e0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:32:36.464344 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 9025
I20260928 06:32:36.464466 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/minicluster-data/master-0/wal/instance
I20260928 06:32:36.464565 9093 raft_consensus.cc:504] T 00000000000000000000000000000000 P e598493c02424281bc5e6fada64030e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e598493c02424281bc5e6fada64030e0" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 38821 } }
I20260928 06:32:36.464749 23430 external_mini_cluster.cc:1428] Setting key 91bfd60e7a547750cbc30272f5d240c6
I20260928 06:32:36.464736 9093 leader_election.cc:302] T 00000000000000000000000000000000 P e598493c02424281bc5e6fada64030e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e598493c02424281bc5e6fada64030e0; no voters:
I20260928 06:32:36.464983 9096 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e598493c02424281bc5e6fada64030e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:32:36.465171 9096 raft_consensus.cc:686] T 00000000000000000000000000000000 P e598493c02424281bc5e6fada64030e0 [term 1 LEADER]: Becoming Leader. State: Replica: e598493c02424281bc5e6fada64030e0, State: Running, Role: LEADER
I20260928 06:32:36.465363 9096 consensus_queue.cc:237] T 00000000000000000000000000000000 P e598493c02424281bc5e6fada64030e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e598493c02424281bc5e6fada64030e0" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 38821 } }
I20260928 06:32:36.465598 9093 leader_election.cc:288] T 00000000000000000000000000000000 P e598493c02424281bc5e6fada64030e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:32:36.465723 9096 sys_catalog.cc:455] T 00000000000000000000000000000000 P e598493c02424281bc5e6fada64030e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e598493c02424281bc5e6fada64030e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e598493c02424281bc5e6fada64030e0" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 38821 } } }
I20260928 06:32:36.465884 9093 sys_catalog.cc:564] T 00000000000000000000000000000000 P e598493c02424281bc5e6fada64030e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:32:36.465895 9097 sys_catalog.cc:455] T 00000000000000000000000000000000 P e598493c02424281bc5e6fada64030e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e598493c02424281bc5e6fada64030e0. Latest consensus state: current_term: 1 leader_uuid: "e598493c02424281bc5e6fada64030e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e598493c02424281bc5e6fada64030e0" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 38821 } } }
I20260928 06:32:36.466024 9097 sys_catalog.cc:458] T 00000000000000000000000000000000 P e598493c02424281bc5e6fada64030e0 [sys.catalog]: This master's current role is: LEADER
I20260928 06:32:36.465888 9096 sys_catalog.cc:458] T 00000000000000000000000000000000 P e598493c02424281bc5e6fada64030e0 [sys.catalog]: This master's current role is: LEADER
I20260928 06:32:36.466632 9102 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:32:36.467188 9102 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:32:36.468914 9102 catalog_manager.cc:1347] Generated new cluster ID: 5a3ffe32b98648e0843aec4df25ed7da
I20260928 06:32:36.468957 9102 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:32:36 dist-test-slave-d59m krb5kdc[9012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:32:36.479597 9099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:36.465458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38374 (local address 127.22.225.190:38821)
0928 06:32:36.465621 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:36.465631 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:36.465666 (+ 35us) server_negotiation.cc:407] Connection header received
0928 06:32:36.465809 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:36.465816 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:36.465874 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:36.466915 (+ 1041us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:32:36.467038 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:36.468467 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:36.469554 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:36.470242 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:36.470657 (+ 415us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:36.475242 (+ 4585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:32:36.475247 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:32:36.475248 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:32:36.475261 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:32:36.477485 (+ 2224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:32:36.477893 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:32:36.477896 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:32:36.477897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:32:36.478005 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:32:36.478210 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:32:36.478211 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:32:36.478211 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:32:36.478542 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:32:36.479117 (+ 575us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:36.479204 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 06:32:36.479345 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":76,"threads_started":1}
I20260928 06:32:36.501844 9102 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:32:36.502637 9102 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:32:36.510353 9102 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e598493c02424281bc5e6fada64030e0: Generated new TSK 0
I20260928 06:32:36.510551 9102 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.225.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab.
Entry for principal kudu/127.22.225.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab.
I20260928 06:32:36.544200 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.129:0
--local_ip_for_outbound_sockets=127.22.225.129
--webserver_interface=127.22.225.129
--webserver_port=0
--tserver_master_addrs=127.22.225.190:38821
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab
--principal=kudu/127.22.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:40447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:32:36.660851 9119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:32:36.661063 9119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:32:36.661088 9119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:32:36.662299 9119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:32:36.662343 9119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:32:36.662369 9119 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.129
I20260928 06:32:36.663928 9119 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:40447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab
--principal=kudu/127.22.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.225.129
--webserver_port=0
--tserver_master_addrs=127.22.225.190:38821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.129
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:32:36.664197 9119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:32:36.664448 9119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:32:36.669615 9125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:36.669606 9127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:36.669811 9124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:32:36.670305 9119 server_base.cc:1034] running on GCE node
I20260928 06:32:36.670524 9119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:32:36.670799 9119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:32:36.671959 9119 hybrid_clock.cc:648] HybridClock initialized: now 1790577156671933 us; error 42 us; skew 500 ppm
Sep 28 06:32:36 dist-test-slave-d59m krb5kdc[9012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577156, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:32:36.674903 9119 init.cc:377] Logged in from keytab as kudu/127.22.225.129@KRBTEST.COM (short username kudu)
I20260928 06:32:36.675395 9119 webserver.cc:460] Webserver started at http://127.22.225.129:39691/ using document root <none> and password file <none>
I20260928 06:32:36.675648 9119 fs_manager.cc:359] Metadata directory not provided
I20260928 06:32:36.675725 9119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:32:36.675892 9119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:32:36.677034 9119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/minicluster-data/ts-0/data/instance:
uuid: "6aaab91e415d495abc7afb337a92f8cb"
format_stamp: "Formatted at 2026-09-28 06:32:36 on dist-test-slave-d59m"
server_key: "8539347d40ddce498e1ac0310a1e4a47"
server_key_iv: "b844602552975bf7d6138ee667379aac"
server_key_version: "encryptionkey@0"
I20260928 06:32:36.677398 9119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/minicluster-data/ts-0/wal/instance:
uuid: "6aaab91e415d495abc7afb337a92f8cb"
format_stamp: "Formatted at 2026-09-28 06:32:36 on dist-test-slave-d59m"
server_key: "8539347d40ddce498e1ac0310a1e4a47"
server_key_iv: "b844602552975bf7d6138ee667379aac"
server_key_version: "encryptionkey@0"
I20260928 06:32:36.678785 9119 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:32:36.679561 9134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:36.679750 9119 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:32:36.679859 9119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/minicluster-data/ts-0/wal
uuid: "6aaab91e415d495abc7afb337a92f8cb"
format_stamp: "Formatted at 2026-09-28 06:32:36 on dist-test-slave-d59m"
server_key: "8539347d40ddce498e1ac0310a1e4a47"
server_key_iv: "b844602552975bf7d6138ee667379aac"
server_key_version: "encryptionkey@0"
I20260928 06:32:36.679929 9119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:32:36.705427 9119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:32:36.707901 9119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:32:36.708078 9119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:32:36.708340 9119 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:32:36.708730 9119 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:32:36.708786 9119 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:36.708832 9119 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:32:36.708858 9119 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:36.715917 9119 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.129:43187
I20260928 06:32:36.716002 9247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.129:43187 every 8 connection(s)
I20260928 06:32:36.716354 9119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
I20260928 06:32:36.720522 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 9119
I20260928 06:32:36.720669 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/minicluster-data/ts-0/wal/instance
I20260928 06:32:36.720882 23430 external_mini_cluster.cc:1428] Setting key af131e576af7e463a430ea1b2034606d
Sep 28 06:32:36 dist-test-slave-d59m krb5kdc[9012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577156, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.129@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:32:36.727698 9099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:36.717205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:48849 (local address 127.22.225.190:38821)
0928 06:32:36.717276 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:36.717278 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:36.717459 (+ 181us) server_negotiation.cc:407] Connection header received
0928 06:32:36.717987 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:36.717989 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:36.718001 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:36.718060 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:32:36.719080 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:36.719600 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:36.720655 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:36.720874 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:36.720964 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:36.724427 (+ 3463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:32:36.724429 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:32:36.724429 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:32:36.724440 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:32:36.726198 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:32:36.726673 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:32:36.726674 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:32:36.726675 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:32:36.726733 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:32:36.726980 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:32:36.726981 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:32:36.726981 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:32:36.727108 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:32:36.727147 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:36.727516 (+ 369us) server_negotiation.cc:299] Negotiation successful
0928 06:32:36.727600 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 06:32:36.728219 9250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:36.717159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:38821 (local address 127.22.225.129:48849)
0928 06:32:36.717324 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:36.717369 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 06:32:36.717869 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:32:36.718138 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:32:36.718148 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:32:36.718750 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:32:36.718973 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:36.718978 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:36.719715 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:36.719716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:36.720527 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:36.720529 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:36.721054 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:36.721056 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:36.721251 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:36.721959 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:32:36.721970 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:32:36.724281 (+ 2311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:32:36.726301 (+ 2020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:32:36.726309 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:32:36.726312 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:32:36.726590 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:32:36.726799 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:32:36.726800 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:32:36.726800 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:32:36.726917 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:32:36.727187 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:32:36.727192 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:32:36.727372 (+ 180us) client_negotiation.cc:769] Sending connection context
0928 06:32:36.727803 (+ 431us) client_negotiation.cc:240] Negotiation successful
0928 06:32:36.727981 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":91,"threads_started":1}
I20260928 06:32:36.728866 9248 heartbeater.cc:347] Connected to a master server at 127.22.225.190:38821
I20260928 06:32:36.728999 9248 heartbeater.cc:464] Registering TS with master...
I20260928 06:32:36.729229 9248 heartbeater.cc:511] Master 127.22.225.190:38821 requested a full tablet report, sending...
I20260928 06:32:36.729775 9057 ts_manager.cc:194] Registered new tserver with Master: 6aaab91e415d495abc7afb337a92f8cb (127.22.225.129:43187)
WARNING: no policy specified for kudu/127.22.225.130@KRBTEST.COM; defaulting to no policy
I20260928 06:32:36.730692 9057 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.225.129@KRBTEST.COM'} at 127.22.225.129:48849
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab.
Entry for principal kudu/127.22.225.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab.
I20260928 06:32:36.747965 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.130:0
--local_ip_for_outbound_sockets=127.22.225.130
--webserver_interface=127.22.225.130
--webserver_port=0
--tserver_master_addrs=127.22.225.190:38821
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab
--principal=kudu/127.22.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:40447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:32:36.840986 9253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:32:36.841198 9253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:32:36.841223 9253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:32:36.842550 9253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:32:36.842592 9253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:32:36.842619 9253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.130
I20260928 06:32:36.844272 9253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:40447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab
--principal=kudu/127.22.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.225.130
--webserver_port=0
--tserver_master_addrs=127.22.225.190:38821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.130
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:32:36.844527 9253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:32:36.844877 9253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:32:36.850394 9259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:36.850425 9258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:36.850526 9261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:32:36.850576 9253 server_base.cc:1034] running on GCE node
I20260928 06:32:36.850829 9253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:32:36.851111 9253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:32:36.852274 9253 hybrid_clock.cc:648] HybridClock initialized: now 1790577156852256 us; error 31 us; skew 500 ppm
Sep 28 06:32:36 dist-test-slave-d59m krb5kdc[9012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577156, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:32:36.855129 9253 init.cc:377] Logged in from keytab as kudu/127.22.225.130@KRBTEST.COM (short username kudu)
I20260928 06:32:36.855625 9253 webserver.cc:460] Webserver started at http://127.22.225.130:43081/ using document root <none> and password file <none>
I20260928 06:32:36.855880 9253 fs_manager.cc:359] Metadata directory not provided
I20260928 06:32:36.855962 9253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:32:36.856122 9253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:32:36.857251 9253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/minicluster-data/ts-1/data/instance:
uuid: "edba5cd1e6ea4169bd9862453a79a35b"
format_stamp: "Formatted at 2026-09-28 06:32:36 on dist-test-slave-d59m"
server_key: "f85e5b14a069957f94795ecae7505d60"
server_key_iv: "3d04265e9cff782751b8700161ea130f"
server_key_version: "encryptionkey@0"
I20260928 06:32:36.857610 9253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/minicluster-data/ts-1/wal/instance:
uuid: "edba5cd1e6ea4169bd9862453a79a35b"
format_stamp: "Formatted at 2026-09-28 06:32:36 on dist-test-slave-d59m"
server_key: "f85e5b14a069957f94795ecae7505d60"
server_key_iv: "3d04265e9cff782751b8700161ea130f"
server_key_version: "encryptionkey@0"
I20260928 06:32:36.858991 9253 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:32:36.859848 9268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:36.860036 9253 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:32:36.860126 9253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/minicluster-data/ts-1/wal
uuid: "edba5cd1e6ea4169bd9862453a79a35b"
format_stamp: "Formatted at 2026-09-28 06:32:36 on dist-test-slave-d59m"
server_key: "f85e5b14a069957f94795ecae7505d60"
server_key_iv: "3d04265e9cff782751b8700161ea130f"
server_key_version: "encryptionkey@0"
I20260928 06:32:36.860219 9253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:32:36.876062 9253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:32:36.878353 9253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:32:36.878496 9253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:32:36.878747 9253 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:32:36.879173 9253 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:32:36.879235 9253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:36.879280 9253 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:32:36.879307 9253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:36.886353 9253 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.130:46187
I20260928 06:32:36.886387 9381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.130:46187 every 8 connection(s)
I20260928 06:32:36.886734 9253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
Sep 28 06:32:36 dist-test-slave-d59m krb5kdc[9012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577156, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.130@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:32:36.893237 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 9253
I20260928 06:32:36.893376 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/minicluster-data/ts-1/wal/instance
I20260928 06:32:36.893571 23430 external_mini_cluster.cc:1428] Setting key d274713e8a43bf55be5374e0cd7a774a
I20260928 06:32:36.897866 9099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:36.887711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:53921 (local address 127.22.225.190:38821)
0928 06:32:36.887786 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:36.887788 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:36.887800 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:32:36.888331 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:36.888333 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:36.888349 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:36.888401 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:32:36.889442 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:36.889952 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:36.890974 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:36.891205 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:36.891296 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:36.893638 (+ 2342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:32:36.893640 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:32:36.893640 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:32:36.893655 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:32:36.896417 (+ 2762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:32:36.896885 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:32:36.896887 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:32:36.896887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:32:36.896949 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:32:36.897217 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:32:36.897218 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:32:36.897219 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:32:36.897336 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:32:36.897407 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:36.897697 (+ 290us) server_negotiation.cc:299] Negotiation successful
0928 06:32:36.897778 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 06:32:36.898202 9384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:36.887481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:38821 (local address 127.22.225.130:53921)
0928 06:32:36.887640 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:36.887677 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 06:32:36.888213 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:32:36.888471 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:32:36.888475 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:32:36.889129 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:32:36.889340 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:36.889346 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:36.890054 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:36.890055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:36.890879 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:36.890881 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:36.891307 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:36.891308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:36.891435 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:36.891937 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:32:36.891945 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:32:36.893538 (+ 1593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:32:36.896508 (+ 2970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:32:36.896515 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:32:36.896518 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:32:36.896792 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:32:36.897017 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:32:36.897018 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:32:36.897018 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:32:36.897134 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:32:36.897410 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:32:36.897418 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:32:36.897556 (+ 138us) client_negotiation.cc:769] Sending connection context
0928 06:32:36.897656 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 06:32:36.897813 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":78,"threads_started":1}
I20260928 06:32:36.898910 9382 heartbeater.cc:347] Connected to a master server at 127.22.225.190:38821
I20260928 06:32:36.899047 9382 heartbeater.cc:464] Registering TS with master...
I20260928 06:32:36.899236 9382 heartbeater.cc:511] Master 127.22.225.190:38821 requested a full tablet report, sending...
I20260928 06:32:36.899621 9057 ts_manager.cc:194] Registered new tserver with Master: edba5cd1e6ea4169bd9862453a79a35b (127.22.225.130:46187)
I20260928 06:32:36.900012 9057 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.225.130@KRBTEST.COM'} at 127.22.225.130:53921
WARNING: no policy specified for kudu/127.22.225.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab.
Entry for principal kudu/127.22.225.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab.
I20260928 06:32:36.921253 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.131:0
--local_ip_for_outbound_sockets=127.22.225.131
--webserver_interface=127.22.225.131
--webserver_port=0
--tserver_master_addrs=127.22.225.190:38821
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab
--principal=kudu/127.22.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:40447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:32:37.036247 9387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:32:37.036545 9387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:32:37.036600 9387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:32:37.038295 9387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:32:37.038348 9387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:32:37.038386 9387 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.131
I20260928 06:32:37.040499 9387 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:40447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab
--principal=kudu/127.22.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.225.131
--webserver_port=0
--tserver_master_addrs=127.22.225.190:38821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9387
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.131
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:32:37.040803 9387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:32:37.041060 9387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:32:37.046772 9392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:37.046614 9393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:37.046617 9395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:32:37.046976 9387 server_base.cc:1034] running on GCE node
I20260928 06:32:37.047178 9387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:32:37.047456 9387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:32:37.048628 9387 hybrid_clock.cc:648] HybridClock initialized: now 1790577157048615 us; error 28 us; skew 500 ppm
Sep 28 06:32:37 dist-test-slave-d59m krb5kdc[9012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577157, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:32:37.051574 9387 init.cc:377] Logged in from keytab as kudu/127.22.225.131@KRBTEST.COM (short username kudu)
I20260928 06:32:37.052110 9387 webserver.cc:460] Webserver started at http://127.22.225.131:40287/ using document root <none> and password file <none>
I20260928 06:32:37.052371 9387 fs_manager.cc:359] Metadata directory not provided
I20260928 06:32:37.052444 9387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:32:37.052618 9387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:32:37.053714 9387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/minicluster-data/ts-2/data/instance:
uuid: "a1253bf77f9b4ded9f4332719eb16819"
format_stamp: "Formatted at 2026-09-28 06:32:37 on dist-test-slave-d59m"
server_key: "c74ea334e9a8768ab53afb1876501760"
server_key_iv: "91870fdb36b679cf6ccfacae5d3738b7"
server_key_version: "encryptionkey@0"
I20260928 06:32:37.054095 9387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/minicluster-data/ts-2/wal/instance:
uuid: "a1253bf77f9b4ded9f4332719eb16819"
format_stamp: "Formatted at 2026-09-28 06:32:37 on dist-test-slave-d59m"
server_key: "c74ea334e9a8768ab53afb1876501760"
server_key_iv: "91870fdb36b679cf6ccfacae5d3738b7"
server_key_version: "encryptionkey@0"
I20260928 06:32:37.055521 9387 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:32:37.056648 9402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:37.056834 9387 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:32:37.056937 9387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/minicluster-data/ts-2/wal
uuid: "a1253bf77f9b4ded9f4332719eb16819"
format_stamp: "Formatted at 2026-09-28 06:32:37 on dist-test-slave-d59m"
server_key: "c74ea334e9a8768ab53afb1876501760"
server_key_iv: "91870fdb36b679cf6ccfacae5d3738b7"
server_key_version: "encryptionkey@0"
I20260928 06:32:37.057018 9387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:32:37.074549 9387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:32:37.077416 9387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:32:37.077572 9387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:32:37.077800 9387 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:32:37.078140 9387 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:32:37.078179 9387 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:37.078218 9387 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:32:37.078235 9387 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:37.085229 9387 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.131:35595
I20260928 06:32:37.085299 9515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.131:35595 every 8 connection(s)
I20260928 06:32:37.085697 9387 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
I20260928 06:32:37.087476 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 9387
I20260928 06:32:37.087574 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/minicluster-data/ts-2/wal/instance
I20260928 06:32:37.087776 23430 external_mini_cluster.cc:1428] Setting key ed64891ec3825ca09f10d1325c7a3d4a
Sep 28 06:32:37 dist-test-slave-d59m krb5kdc[9012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577157, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.131@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:32:37.095952 9099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:37.086552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:48595 (local address 127.22.225.190:38821)
0928 06:32:37.086619 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:37.086622 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:37.086870 (+ 248us) server_negotiation.cc:407] Connection header received
0928 06:32:37.087401 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:37.087402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:37.087414 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:37.087490 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:32:37.088852 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:37.089406 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:37.090449 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:37.090675 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:37.090756 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:37.093007 (+ 2251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:32:37.093009 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:32:37.093009 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:32:37.093017 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:32:37.094501 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:32:37.094934 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:32:37.094935 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:32:37.094936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:32:37.094998 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:32:37.095273 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:32:37.095274 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:32:37.095274 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:32:37.095394 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:32:37.095469 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:37.095765 (+ 296us) server_negotiation.cc:299] Negotiation successful
0928 06:32:37.095859 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 06:32:37.096171 9518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:37.086541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:38821 (local address 127.22.225.131:48595)
0928 06:32:37.086731 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:37.086776 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 06:32:37.087288 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:32:37.087575 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:32:37.087580 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:32:37.088432 (+ 852us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:32:37.088744 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:37.088751 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:37.089517 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:37.089518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:37.090358 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:37.090360 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:37.090762 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:37.090763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:37.090898 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:37.091477 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:32:37.091485 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:32:37.092906 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:32:37.094607 (+ 1701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:32:37.094615 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:32:37.094617 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:32:37.094851 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:32:37.095069 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:32:37.095070 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:32:37.095070 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:32:37.095196 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:32:37.095471 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:32:37.095479 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:32:37.095624 (+ 145us) client_negotiation.cc:769] Sending connection context
0928 06:32:37.095742 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 06:32:37.095907 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":116,"threads_started":1}
I20260928 06:32:37.096662 9516 heartbeater.cc:347] Connected to a master server at 127.22.225.190:38821
I20260928 06:32:37.096768 9516 heartbeater.cc:464] Registering TS with master...
I20260928 06:32:37.096958 9516 heartbeater.cc:511] Master 127.22.225.190:38821 requested a full tablet report, sending...
I20260928 06:32:37.097385 9057 ts_manager.cc:194] Registered new tserver with Master: a1253bf77f9b4ded9f4332719eb16819 (127.22.225.131:35595)
I20260928 06:32:37.097787 9057 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.225.131@KRBTEST.COM'} at 127.22.225.131:48595
I20260928 06:32:37.099830 23430 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:32:37.107887 9099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:37.101348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38386 (local address 127.22.225.190:38821)
0928 06:32:37.101394 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:37.101395 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:37.101405 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:32:37.101474 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:37.101475 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:37.101486 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:37.101541 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:32:37.102094 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:37.102574 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:37.103326 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:37.103536 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:37.103616 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:37.105092 (+ 1476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:32:37.105093 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:32:37.105093 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:32:37.105101 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:32:37.106581 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:32:37.107012 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:32:37.107014 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:32:37.107014 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:32:37.107072 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:32:37.107314 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:32:37.107316 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:32:37.107316 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:32:37.107471 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:32:37.107588 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:37.107720 (+ 132us) server_negotiation.cc:299] Negotiation successful
0928 06:32:37.107785 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 06:32:37.109716 9057 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38386:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:32:37.111549 9057 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:32:37.119028 9532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:37.114234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60050 (local address 127.22.225.129:43187)
0928 06:32:37.114562 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:37.114566 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:37.114575 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:32:37.114616 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:37.114618 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:37.114634 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:37.114710 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:32:37.114933 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:37.116106 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:37.117785 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:37.118530 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:37.118586 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:37.118647 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:37.118904 (+ 257us) server_negotiation.cc:299] Negotiation successful
0928 06:32:37.118946 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":98,"threads_started":1}
I20260928 06:32:37.119004 9528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:37.113958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.130:46187 (local address 127.0.0.1:58866)
0928 06:32:37.114512 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:37.114518 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0928 06:32:37.114581 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:32:37.114826 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:32:37.114827 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:32:37.114828 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:32:37.114861 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:37.114863 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:37.116189 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:37.116191 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:37.117663 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:37.117666 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:37.118685 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:37.118686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:37.118771 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:37.118788 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 06:32:37.118857 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 06:32:37.118918 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"mutex_wait_us":67,"thread_start_us":106,"threads_started":1}
I20260928 06:32:37.119240 9527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:37.113954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.129:43187 (local address 127.0.0.1:60050)
0928 06:32:37.114067 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:37.114110 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:32:37.114181 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:32:37.114734 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:32:37.114737 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:32:37.114739 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:32:37.114827 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:37.114829 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:37.116217 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:37.116217 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:37.117663 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:37.117666 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:37.118718 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:37.118719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:37.118771 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:37.118788 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 06:32:37.119114 (+ 326us) client_negotiation.cc:240] Negotiation successful
0928 06:32:37.119170 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"mutex_wait_us":60,"thread_start_us":63,"threads_started":1}
I20260928 06:32:37.119803 9529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:37.113958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.131:35595 (local address 127.0.0.1:35480)
0928 06:32:37.114388 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:37.114398 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:32:37.114456 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:32:37.115382 (+ 926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:32:37.115382 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:32:37.115384 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:32:37.115428 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:37.115429 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:37.117043 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:37.117044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:37.118260 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:37.118263 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:37.119575 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:37.119576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:37.119640 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:37.119644 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:32:37.119687 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 06:32:37.119731 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"mutex_wait_us":1,"thread_start_us":107,"threads_started":1}
I20260928 06:32:37.119851 9531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:37.114222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58866 (local address 127.22.225.130:46187)
0928 06:32:37.114485 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:37.114491 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:37.114585 (+ 94us) server_negotiation.cc:407] Connection header received
0928 06:32:37.114656 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:37.114659 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:37.114674 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:37.114748 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:32:37.114944 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:37.116062 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:37.117773 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:37.118530 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:37.118601 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:37.118647 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:37.119715 (+ 1068us) server_negotiation.cc:299] Negotiation successful
0928 06:32:37.119768 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":86,"threads_started":1}
I20260928 06:32:37.120318 9530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:37.114218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35480 (local address 127.22.225.131:35595)
0928 06:32:37.115155 (+ 937us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:37.115160 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:37.115169 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:32:37.115206 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:37.115208 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:37.115234 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:37.115303 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:32:37.115495 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:37.116633 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:37.118353 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:37.119373 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:37.119496 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:37.119553 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:37.120183 (+ 630us) server_negotiation.cc:299] Negotiation successful
0928 06:32:37.120239 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":896,"thread_start_us":87,"threads_started":1}
I20260928 06:32:37.120551 9316 tablet_service.cc:1467] Processing CreateTablet for tablet 282659f3cf0b4b579fc86b03cca7eebd (DEFAULT_TABLE table=test-table [id=1cdf65976ee3475baab368d8ab867852]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:32:37.120553 9182 tablet_service.cc:1467] Processing CreateTablet for tablet 282659f3cf0b4b579fc86b03cca7eebd (DEFAULT_TABLE table=test-table [id=1cdf65976ee3475baab368d8ab867852]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:32:37.120985 9316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 282659f3cf0b4b579fc86b03cca7eebd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:32:37.120985 9182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 282659f3cf0b4b579fc86b03cca7eebd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:32:37.124542 9533 tablet_bootstrap.cc:492] T 282659f3cf0b4b579fc86b03cca7eebd P 6aaab91e415d495abc7afb337a92f8cb: Bootstrap starting.
I20260928 06:32:37.125026 9450 tablet_service.cc:1467] Processing CreateTablet for tablet 282659f3cf0b4b579fc86b03cca7eebd (DEFAULT_TABLE table=test-table [id=1cdf65976ee3475baab368d8ab867852]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:32:37.125299 9533 tablet_bootstrap.cc:654] T 282659f3cf0b4b579fc86b03cca7eebd P 6aaab91e415d495abc7afb337a92f8cb: Neither blocks nor log segments found. Creating new log.
I20260928 06:32:37.125401 9450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 282659f3cf0b4b579fc86b03cca7eebd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:32:37.125910 9533 log.cc:824] T 282659f3cf0b4b579fc86b03cca7eebd P 6aaab91e415d495abc7afb337a92f8cb: Log is configured to *not* fsync() on all Append() calls
I20260928 06:32:37.126257 9534 tablet_bootstrap.cc:492] T 282659f3cf0b4b579fc86b03cca7eebd P edba5cd1e6ea4169bd9862453a79a35b: Bootstrap starting.
I20260928 06:32:37.126888 9534 tablet_bootstrap.cc:654] T 282659f3cf0b4b579fc86b03cca7eebd P edba5cd1e6ea4169bd9862453a79a35b: Neither blocks nor log segments found. Creating new log.
I20260928 06:32:37.127259 9533 tablet_bootstrap.cc:492] T 282659f3cf0b4b579fc86b03cca7eebd P 6aaab91e415d495abc7afb337a92f8cb: No bootstrap required, opened a new log
I20260928 06:32:37.127329 9533 ts_tablet_manager.cc:1397] T 282659f3cf0b4b579fc86b03cca7eebd P 6aaab91e415d495abc7afb337a92f8cb: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260928 06:32:37.128465 9536 tablet_bootstrap.cc:492] T 282659f3cf0b4b579fc86b03cca7eebd P a1253bf77f9b4ded9f4332719eb16819: Bootstrap starting.
I20260928 06:32:37.129156 9536 tablet_bootstrap.cc:654] T 282659f3cf0b4b579fc86b03cca7eebd P a1253bf77f9b4ded9f4332719eb16819: Neither blocks nor log segments found. Creating new log.
I20260928 06:32:37.129204 9533 raft_consensus.cc:348] T 282659f3cf0b4b579fc86b03cca7eebd P 6aaab91e415d495abc7afb337a92f8cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aaab91e415d495abc7afb337a92f8cb" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 43187 } } peers { permanent_uuid: "edba5cd1e6ea4169bd9862453a79a35b" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46187 } } peers { permanent_uuid: "a1253bf77f9b4ded9f4332719eb16819" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 35595 } }
I20260928 06:32:37.129412 9533 raft_consensus.cc:374] T 282659f3cf0b4b579fc86b03cca7eebd P 6aaab91e415d495abc7afb337a92f8cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:32:37.129460 9533 raft_consensus.cc:729] T 282659f3cf0b4b579fc86b03cca7eebd P 6aaab91e415d495abc7afb337a92f8cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6aaab91e415d495abc7afb337a92f8cb, State: Initialized, Role: FOLLOWER
I20260928 06:32:37.129616 9533 consensus_queue.cc:260] T 282659f3cf0b4b579fc86b03cca7eebd P 6aaab91e415d495abc7afb337a92f8cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aaab91e415d495abc7afb337a92f8cb" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 43187 } } peers { permanent_uuid: "edba5cd1e6ea4169bd9862453a79a35b" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46187 } } peers { permanent_uuid: "a1253bf77f9b4ded9f4332719eb16819" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 35595 } }
I20260928 06:32:37.129645 9536 log.cc:824] T 282659f3cf0b4b579fc86b03cca7eebd P a1253bf77f9b4ded9f4332719eb16819: Log is configured to *not* fsync() on all Append() calls
I20260928 06:32:37.129909 9533 ts_tablet_manager.cc:1428] T 282659f3cf0b4b579fc86b03cca7eebd P 6aaab91e415d495abc7afb337a92f8cb: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260928 06:32:37.129973 9248 heartbeater.cc:503] Master 127.22.225.190:38821 was elected leader, sending a full tablet report...
I20260928 06:32:37.130573 9534 log.cc:824] T 282659f3cf0b4b579fc86b03cca7eebd P edba5cd1e6ea4169bd9862453a79a35b: Log is configured to *not* fsync() on all Append() calls
I20260928 06:32:37.130879 9536 tablet_bootstrap.cc:492] T 282659f3cf0b4b579fc86b03cca7eebd P a1253bf77f9b4ded9f4332719eb16819: No bootstrap required, opened a new log
I20260928 06:32:37.130957 9536 ts_tablet_manager.cc:1397] T 282659f3cf0b4b579fc86b03cca7eebd P a1253bf77f9b4ded9f4332719eb16819: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260928 06:32:37.132748 9536 raft_consensus.cc:348] T 282659f3cf0b4b579fc86b03cca7eebd P a1253bf77f9b4ded9f4332719eb16819 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aaab91e415d495abc7afb337a92f8cb" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 43187 } } peers { permanent_uuid: "edba5cd1e6ea4169bd9862453a79a35b" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46187 } } peers { permanent_uuid: "a1253bf77f9b4ded9f4332719eb16819" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 35595 } }
I20260928 06:32:37.132965 9536 raft_consensus.cc:374] T 282659f3cf0b4b579fc86b03cca7eebd P a1253bf77f9b4ded9f4332719eb16819 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:32:37.133028 9536 raft_consensus.cc:729] T 282659f3cf0b4b579fc86b03cca7eebd P a1253bf77f9b4ded9f4332719eb16819 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1253bf77f9b4ded9f4332719eb16819, State: Initialized, Role: FOLLOWER
I20260928 06:32:37.133152 9536 consensus_queue.cc:260] T 282659f3cf0b4b579fc86b03cca7eebd P a1253bf77f9b4ded9f4332719eb16819 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aaab91e415d495abc7afb337a92f8cb" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 43187 } } peers { permanent_uuid: "edba5cd1e6ea4169bd9862453a79a35b" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46187 } } peers { permanent_uuid: "a1253bf77f9b4ded9f4332719eb16819" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 35595 } }
I20260928 06:32:37.133423 9536 ts_tablet_manager.cc:1428] T 282659f3cf0b4b579fc86b03cca7eebd P a1253bf77f9b4ded9f4332719eb16819: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:32:37.133457 9516 heartbeater.cc:503] Master 127.22.225.190:38821 was elected leader, sending a full tablet report...
I20260928 06:32:37.134182 9534 tablet_bootstrap.cc:492] T 282659f3cf0b4b579fc86b03cca7eebd P edba5cd1e6ea4169bd9862453a79a35b: No bootstrap required, opened a new log
I20260928 06:32:37.134270 9534 ts_tablet_manager.cc:1397] T 282659f3cf0b4b579fc86b03cca7eebd P edba5cd1e6ea4169bd9862453a79a35b: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260928 06:32:37.135720 9534 raft_consensus.cc:348] T 282659f3cf0b4b579fc86b03cca7eebd P edba5cd1e6ea4169bd9862453a79a35b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aaab91e415d495abc7afb337a92f8cb" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 43187 } } peers { permanent_uuid: "edba5cd1e6ea4169bd9862453a79a35b" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46187 } } peers { permanent_uuid: "a1253bf77f9b4ded9f4332719eb16819" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 35595 } }
I20260928 06:32:37.135878 9534 raft_consensus.cc:374] T 282659f3cf0b4b579fc86b03cca7eebd P edba5cd1e6ea4169bd9862453a79a35b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:32:37.135943 9534 raft_consensus.cc:729] T 282659f3cf0b4b579fc86b03cca7eebd P edba5cd1e6ea4169bd9862453a79a35b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edba5cd1e6ea4169bd9862453a79a35b, State: Initialized, Role: FOLLOWER
I20260928 06:32:37.136063 9534 consensus_queue.cc:260] T 282659f3cf0b4b579fc86b03cca7eebd P edba5cd1e6ea4169bd9862453a79a35b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aaab91e415d495abc7afb337a92f8cb" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 43187 } } peers { permanent_uuid: "edba5cd1e6ea4169bd9862453a79a35b" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46187 } } peers { permanent_uuid: "a1253bf77f9b4ded9f4332719eb16819" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 35595 } }
I20260928 06:32:37.136360 9534 ts_tablet_manager.cc:1428] T 282659f3cf0b4b579fc86b03cca7eebd P edba5cd1e6ea4169bd9862453a79a35b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:32:37.136451 9382 heartbeater.cc:503] Master 127.22.225.190:38821 was elected leader, sending a full tablet report...
W20260928 06:32:37.137943 9383 tablet.cc:2369] T 282659f3cf0b4b579fc86b03cca7eebd P edba5cd1e6ea4169bd9862453a79a35b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:32:37.217262 9249 tablet.cc:2369] T 282659f3cf0b4b579fc86b03cca7eebd P 6aaab91e415d495abc7afb337a92f8cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:32:37.258174 9537 raft_consensus.cc:482] T 282659f3cf0b4b579fc86b03cca7eebd P 6aaab91e415d495abc7afb337a92f8cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:32:37.258296 9537 raft_consensus.cc:504] T 282659f3cf0b4b579fc86b03cca7eebd P 6aaab91e415d495abc7afb337a92f8cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aaab91e415d495abc7afb337a92f8cb" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 43187 } } peers { permanent_uuid: "edba5cd1e6ea4169bd9862453a79a35b" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46187 } } peers { permanent_uuid: "a1253bf77f9b4ded9f4332719eb16819" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 35595 } }
I20260928 06:32:37.258723 9537 leader_election.cc:288] T 282659f3cf0b4b579fc86b03cca7eebd P 6aaab91e415d495abc7afb337a92f8cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers edba5cd1e6ea4169bd9862453a79a35b (127.22.225.130:46187), a1253bf77f9b4ded9f4332719eb16819 (127.22.225.131:35595)
I20260928 06:32:37.262406 9530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:37.259071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:48137 (local address 127.22.225.131:35595)
0928 06:32:37.259120 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:37.259122 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:37.259172 (+ 50us) server_negotiation.cc:407] Connection header received
0928 06:32:37.259286 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:37.259287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:37.259307 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:37.259340 (+ 33us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:32:37.259621 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:37.260340 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:37.261454 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:37.262023 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:37.262128 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:37.262138 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:37.262305 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 06:32:37.262344 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 06:32:37.262422 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:37.258890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.131:35595 (local address 127.22.225.129:48137)
0928 06:32:37.259081 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:37.259093 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:32:37.259203 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:32:37.259421 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:32:37.259421 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:32:37.259423 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:32:37.259492 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:37.259494 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:37.260443 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:37.260444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:37.261356 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:37.261359 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:37.262129 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:37.262130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:37.262215 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:37.262219 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:32:37.262296 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 06:32:37.262335 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":85,"threads_started":1}
I20260928 06:32:37.262745 9531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:37.259088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:33179 (local address 127.22.225.130:46187)
0928 06:32:37.259164 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:37.259166 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:37.259231 (+ 65us) server_negotiation.cc:407] Connection header received
0928 06:32:37.259273 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:37.259274 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:37.259286 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:37.259348 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:32:37.259544 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:37.260340 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:37.261807 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:37.262374 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:37.262417 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:37.262424 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:37.262623 (+ 199us) server_negotiation.cc:299] Negotiation successful
0928 06:32:37.262660 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 06:32:37.262959 9542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:37.258891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.130:46187 (local address 127.22.225.129:33179)
0928 06:32:37.259141 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:37.259150 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:32:37.259221 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:32:37.259370 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:32:37.259371 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:32:37.259373 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:32:37.259461 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:37.259463 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:37.260421 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:37.260422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:37.261698 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:37.261701 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:37.262480 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:37.262481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:37.262553 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:37.262556 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:32:37.262843 (+ 287us) client_negotiation.cc:240] Negotiation successful
0928 06:32:37.262883 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":59,"spinlock_wait_cycles":11648,"thread_start_us":92,"threads_started":1}
I20260928 06:32:37.262987 9470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "282659f3cf0b4b579fc86b03cca7eebd" candidate_uuid: "6aaab91e415d495abc7afb337a92f8cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1253bf77f9b4ded9f4332719eb16819" is_pre_election: true
I20260928 06:32:37.263175 9470 raft_consensus.cc:2454] T 282659f3cf0b4b579fc86b03cca7eebd P a1253bf77f9b4ded9f4332719eb16819 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6aaab91e415d495abc7afb337a92f8cb in term 0.
I20260928 06:32:37.263355 9336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "282659f3cf0b4b579fc86b03cca7eebd" candidate_uuid: "6aaab91e415d495abc7afb337a92f8cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edba5cd1e6ea4169bd9862453a79a35b" is_pre_election: true
I20260928 06:32:37.263461 9137 leader_election.cc:302] T 282659f3cf0b4b579fc86b03cca7eebd P 6aaab91e415d495abc7afb337a92f8cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6aaab91e415d495abc7afb337a92f8cb, a1253bf77f9b4ded9f4332719eb16819; no voters:
I20260928 06:32:37.263516 9336 raft_consensus.cc:2454] T 282659f3cf0b4b579fc86b03cca7eebd P edba5cd1e6ea4169bd9862453a79a35b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6aaab91e415d495abc7afb337a92f8cb in term 0.
I20260928 06:32:37.263597 9537 raft_consensus.cc:2781] T 282659f3cf0b4b579fc86b03cca7eebd P 6aaab91e415d495abc7afb337a92f8cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:32:37.263654 9537 raft_consensus.cc:482] T 282659f3cf0b4b579fc86b03cca7eebd P 6aaab91e415d495abc7afb337a92f8cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:32:37.263679 9537 raft_consensus.cc:3037] T 282659f3cf0b4b579fc86b03cca7eebd P 6aaab91e415d495abc7afb337a92f8cb [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:32:37.264453 9537 raft_consensus.cc:504] T 282659f3cf0b4b579fc86b03cca7eebd P 6aaab91e415d495abc7afb337a92f8cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aaab91e415d495abc7afb337a92f8cb" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 43187 } } peers { permanent_uuid: "edba5cd1e6ea4169bd9862453a79a35b" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46187 } } peers { permanent_uuid: "a1253bf77f9b4ded9f4332719eb16819" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 35595 } }
I20260928 06:32:37.264582 9537 leader_election.cc:288] T 282659f3cf0b4b579fc86b03cca7eebd P 6aaab91e415d495abc7afb337a92f8cb [CANDIDATE]: Term 1 election: Requested vote from peers edba5cd1e6ea4169bd9862453a79a35b (127.22.225.130:46187), a1253bf77f9b4ded9f4332719eb16819 (127.22.225.131:35595)
I20260928 06:32:37.264717 9336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "282659f3cf0b4b579fc86b03cca7eebd" candidate_uuid: "6aaab91e415d495abc7afb337a92f8cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edba5cd1e6ea4169bd9862453a79a35b"
I20260928 06:32:37.264770 9470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "282659f3cf0b4b579fc86b03cca7eebd" candidate_uuid: "6aaab91e415d495abc7afb337a92f8cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1253bf77f9b4ded9f4332719eb16819"
I20260928 06:32:37.264819 9336 raft_consensus.cc:3037] T 282659f3cf0b4b579fc86b03cca7eebd P edba5cd1e6ea4169bd9862453a79a35b [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:32:37.264837 9470 raft_consensus.cc:3037] T 282659f3cf0b4b579fc86b03cca7eebd P a1253bf77f9b4ded9f4332719eb16819 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:32:37.265590 9470 raft_consensus.cc:2454] T 282659f3cf0b4b579fc86b03cca7eebd P a1253bf77f9b4ded9f4332719eb16819 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6aaab91e415d495abc7afb337a92f8cb in term 1.
I20260928 06:32:37.265621 9336 raft_consensus.cc:2454] T 282659f3cf0b4b579fc86b03cca7eebd P edba5cd1e6ea4169bd9862453a79a35b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6aaab91e415d495abc7afb337a92f8cb in term 1.
I20260928 06:32:37.265781 9137 leader_election.cc:302] T 282659f3cf0b4b579fc86b03cca7eebd P 6aaab91e415d495abc7afb337a92f8cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6aaab91e415d495abc7afb337a92f8cb, a1253bf77f9b4ded9f4332719eb16819; no voters:
I20260928 06:32:37.265933 9537 raft_consensus.cc:2781] T 282659f3cf0b4b579fc86b03cca7eebd P 6aaab91e415d495abc7afb337a92f8cb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:32:37.266122 9537 raft_consensus.cc:686] T 282659f3cf0b4b579fc86b03cca7eebd P 6aaab91e415d495abc7afb337a92f8cb [term 1 LEADER]: Becoming Leader. State: Replica: 6aaab91e415d495abc7afb337a92f8cb, State: Running, Role: LEADER
I20260928 06:32:37.266211 9537 consensus_queue.cc:237] T 282659f3cf0b4b579fc86b03cca7eebd P 6aaab91e415d495abc7afb337a92f8cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aaab91e415d495abc7afb337a92f8cb" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 43187 } } peers { permanent_uuid: "edba5cd1e6ea4169bd9862453a79a35b" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46187 } } peers { permanent_uuid: "a1253bf77f9b4ded9f4332719eb16819" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 35595 } }
I20260928 06:32:37.267119 9056 catalog_manager.cc:5495] T 282659f3cf0b4b579fc86b03cca7eebd P 6aaab91e415d495abc7afb337a92f8cb reported cstate change: term changed from 0 to 1, leader changed from <none> to 6aaab91e415d495abc7afb337a92f8cb (127.22.225.129). New cstate: current_term: 1 leader_uuid: "6aaab91e415d495abc7afb337a92f8cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aaab91e415d495abc7afb337a92f8cb" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 43187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "edba5cd1e6ea4169bd9862453a79a35b" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1253bf77f9b4ded9f4332719eb16819" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 35595 } health_report { overall_health: UNKNOWN } } }
I20260928 06:32:37.307823 9532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:37.305041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60052 (local address 127.22.225.129:43187)
0928 06:32:37.305114 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:37.305116 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:37.305125 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:32:37.305172 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:37.305173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:37.305195 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:37.305275 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:32:37.305418 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:37.306176 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:37.306904 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:37.307219 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:37.307297 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:37.307472 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:32:37.307590 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:32:37.307685 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:37.307730 (+ 45us) server_negotiation.cc:299] Negotiation successful
0928 06:32:37.307741 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 06:32:37.309731 9470 raft_consensus.cc:1261] T 282659f3cf0b4b579fc86b03cca7eebd P a1253bf77f9b4ded9f4332719eb16819 [term 1 FOLLOWER]: Refusing update from remote peer 6aaab91e415d495abc7afb337a92f8cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:32:37.309784 9336 raft_consensus.cc:1261] T 282659f3cf0b4b579fc86b03cca7eebd P edba5cd1e6ea4169bd9862453a79a35b [term 1 FOLLOWER]: Refusing update from remote peer 6aaab91e415d495abc7afb337a92f8cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:32:37.310129 9544 consensus_queue.cc:1035] T 282659f3cf0b4b579fc86b03cca7eebd P 6aaab91e415d495abc7afb337a92f8cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1253bf77f9b4ded9f4332719eb16819" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:32:37.310294 9537 consensus_queue.cc:1035] T 282659f3cf0b4b579fc86b03cca7eebd P 6aaab91e415d495abc7afb337a92f8cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "edba5cd1e6ea4169bd9862453a79a35b" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 46187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:32:37.315295 9057 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38386:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:32:37.315392 9057 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38386:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:32:37.316069 9057 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e598493c02424281bc5e6fada64030e0: Sending DeleteTablet for 3 replicas of tablet 282659f3cf0b4b579fc86b03cca7eebd
I20260928 06:32:37.316524 9182 tablet_service.cc:1514] Processing DeleteTablet for tablet 282659f3cf0b4b579fc86b03cca7eebd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:32:37 UTC) from {username='kudu', principal='kudu/127.22.225.190@KRBTEST.COM'} at 127.0.0.1:60050
I20260928 06:32:37.316764 9557 tablet_replica.cc:330] T 282659f3cf0b4b579fc86b03cca7eebd P 6aaab91e415d495abc7afb337a92f8cb: stopping tablet replica
I20260928 06:32:37.316761 9450 tablet_service.cc:1514] Processing DeleteTablet for tablet 282659f3cf0b4b579fc86b03cca7eebd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:32:37 UTC) from {username='kudu', principal='kudu/127.22.225.190@KRBTEST.COM'} at 127.0.0.1:35480
I20260928 06:32:37.316854 9557 raft_consensus.cc:2229] T 282659f3cf0b4b579fc86b03cca7eebd P 6aaab91e415d495abc7afb337a92f8cb [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:32:37.316937 9557 raft_consensus.cc:2258] T 282659f3cf0b4b579fc86b03cca7eebd P 6aaab91e415d495abc7afb337a92f8cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:32:37.316972 9316 tablet_service.cc:1514] Processing DeleteTablet for tablet 282659f3cf0b4b579fc86b03cca7eebd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:32:37 UTC) from {username='kudu', principal='kudu/127.22.225.190@KRBTEST.COM'} at 127.0.0.1:58866
I20260928 06:32:37.317083 9558 tablet_replica.cc:330] T 282659f3cf0b4b579fc86b03cca7eebd P a1253bf77f9b4ded9f4332719eb16819: stopping tablet replica
I20260928 06:32:37.317333 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 9119
I20260928 06:32:37.317587 9557 ts_tablet_manager.cc:1905] T 282659f3cf0b4b579fc86b03cca7eebd P 6aaab91e415d495abc7afb337a92f8cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:32:37.317974 9559 tablet_replica.cc:330] T 282659f3cf0b4b579fc86b03cca7eebd P edba5cd1e6ea4169bd9862453a79a35b: stopping tablet replica
I20260928 06:32:37.318073 9559 raft_consensus.cc:2229] T 282659f3cf0b4b579fc86b03cca7eebd P edba5cd1e6ea4169bd9862453a79a35b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:32:37.318176 9559 raft_consensus.cc:2258] T 282659f3cf0b4b579fc86b03cca7eebd P edba5cd1e6ea4169bd9862453a79a35b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:32:37.318638 9559 ts_tablet_manager.cc:1905] T 282659f3cf0b4b579fc86b03cca7eebd P edba5cd1e6ea4169bd9862453a79a35b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:32:37.320641 9559 ts_tablet_manager.cc:1918] T 282659f3cf0b4b579fc86b03cca7eebd P edba5cd1e6ea4169bd9862453a79a35b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:32:37.320704 9559 log.cc:1196] T 282659f3cf0b4b579fc86b03cca7eebd P edba5cd1e6ea4169bd9862453a79a35b: Deleting WAL directory at /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/minicluster-data/ts-1/wal/wals/282659f3cf0b4b579fc86b03cca7eebd
I20260928 06:32:37.320969 9559 ts_tablet_manager.cc:1939] T 282659f3cf0b4b579fc86b03cca7eebd P edba5cd1e6ea4169bd9862453a79a35b: Deleting consensus metadata
I20260928 06:32:37.321671 9044 catalog_manager.cc:4841] TS edba5cd1e6ea4169bd9862453a79a35b (127.22.225.130:46187): tablet 282659f3cf0b4b579fc86b03cca7eebd (table test-table [id=1cdf65976ee3475baab368d8ab867852]) successfully deleted
I20260928 06:32:37.324697 9558 raft_consensus.cc:2229] T 282659f3cf0b4b579fc86b03cca7eebd P a1253bf77f9b4ded9f4332719eb16819 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:32:37.324879 9558 raft_consensus.cc:2258] T 282659f3cf0b4b579fc86b03cca7eebd P a1253bf77f9b4ded9f4332719eb16819 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:32:37.325307 9558 ts_tablet_manager.cc:1905] T 282659f3cf0b4b579fc86b03cca7eebd P a1253bf77f9b4ded9f4332719eb16819: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 06:32:37.325865 9041 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.225.129:43187 (error 108)
I20260928 06:32:37.326128 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 9253
I20260928 06:32:37.326506 9529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:37.326375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.129:43187 (local address 127.0.0.1:60054)
0928 06:32:37.326434 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:37.326472 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.129:43187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 06:32:37.326619 9041 catalog_manager.cc:4584] TS 6aaab91e415d495abc7afb337a92f8cb (127.22.225.129:43187): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 282659f3cf0b4b579fc86b03cca7eebd: Network error: Client connection negotiation failed: client connection to 127.22.225.129:43187: connect: Connection refused (error 111)
I20260928 06:32:37.327083 9558 ts_tablet_manager.cc:1918] T 282659f3cf0b4b579fc86b03cca7eebd P a1253bf77f9b4ded9f4332719eb16819: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:32:37.327175 9558 log.cc:1196] T 282659f3cf0b4b579fc86b03cca7eebd P a1253bf77f9b4ded9f4332719eb16819: Deleting WAL directory at /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576899143097-23430-0/minicluster-data/ts-2/wal/wals/282659f3cf0b4b579fc86b03cca7eebd
I20260928 06:32:37.327524 9558 ts_tablet_manager.cc:1939] T 282659f3cf0b4b579fc86b03cca7eebd P a1253bf77f9b4ded9f4332719eb16819: Deleting consensus metadata
I20260928 06:32:37.327958 9043 catalog_manager.cc:4841] TS a1253bf77f9b4ded9f4332719eb16819 (127.22.225.131:35595): tablet 282659f3cf0b4b579fc86b03cca7eebd (table test-table [id=1cdf65976ee3475baab368d8ab867852]) successfully deleted
I20260928 06:32:37.332749 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 9387
I20260928 06:32:37.339474 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 9025
2026-09-28T06:32:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2243 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T06:32:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:32:37Z Disabled control of system clock
I20260928 06:32:37.368283 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576899143097-23430-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576899143097-23430-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576899143097-23430-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:43159
--webserver_interface=127.22.225.190
--webserver_port=0
--builtin_ntp_servers=127.22.225.148:46879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.225.190:43159
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 06:32:37.473052 9564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:32:37.473299 9564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:32:37.473336 9564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:32:37.474736 9564 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:32:37.474802 9564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:32:37.474823 9564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:32:37.474857 9564 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:32:37.474884 9564 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:32:37.476480 9564 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:46879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576899143097-23430-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576899143097-23430-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.225.190:43159
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:43159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576899143097-23430-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:32:37.476799 9564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:32:37.477190 9564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:32:37.482074 9569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:37.482122 9570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:37.482079 9572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:32:37.482314 9564 server_base.cc:1034] running on GCE node
I20260928 06:32:37.482730 9564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:32:37.483055 9564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:32:37.484228 9564 hybrid_clock.cc:648] HybridClock initialized: now 1790577157484197 us; error 42 us; skew 500 ppm
I20260928 06:32:37.485666 9564 webserver.cc:460] Webserver started at http://127.22.225.190:43743/ using document root <none> and password file <none>
I20260928 06:32:37.485977 9564 fs_manager.cc:359] Metadata directory not provided
I20260928 06:32:37.486025 9564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:32:37.486143 9564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:32:37.487526 9564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576899143097-23430-0/minicluster-data/master-0/data/instance:
uuid: "dd32aba027bc4668a99a18cd74e41e79"
format_stamp: "Formatted at 2026-09-28 06:32:37 on dist-test-slave-d59m"
server_key: "2983e537292c736db26f6e569f5bbb43"
server_key_iv: "5fe6acf972484385f7f8b382498aae52"
server_key_version: "encryptionkey@0"
I20260928 06:32:37.487943 9564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576899143097-23430-0/minicluster-data/master-0/wal/instance:
uuid: "dd32aba027bc4668a99a18cd74e41e79"
format_stamp: "Formatted at 2026-09-28 06:32:37 on dist-test-slave-d59m"
server_key: "2983e537292c736db26f6e569f5bbb43"
server_key_iv: "5fe6acf972484385f7f8b382498aae52"
server_key_version: "encryptionkey@0"
I20260928 06:32:37.489687 9564 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.002s
I20260928 06:32:37.490552 9578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:37.490724 9564 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:32:37.490821 9564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576899143097-23430-0/minicluster-data/master-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576899143097-23430-0/minicluster-data/master-0/wal
uuid: "dd32aba027bc4668a99a18cd74e41e79"
format_stamp: "Formatted at 2026-09-28 06:32:37 on dist-test-slave-d59m"
server_key: "2983e537292c736db26f6e569f5bbb43"
server_key_iv: "5fe6acf972484385f7f8b382498aae52"
server_key_version: "encryptionkey@0"
I20260928 06:32:37.490896 9564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576899143097-23430-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576899143097-23430-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576899143097-23430-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:32:37.518971 9564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:32:37.519384 9564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:32:37.519536 9564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:32:37.524207 9564 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.190:43159
I20260928 06:32:37.524235 9630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.190:43159 every 8 connection(s)
I20260928 06:32:37.524765 9564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
I20260928 06:32:37.525933 9631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:32:37.529157 9631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd32aba027bc4668a99a18cd74e41e79: Bootstrap starting.
I20260928 06:32:37.529865 9631 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd32aba027bc4668a99a18cd74e41e79: Neither blocks nor log segments found. Creating new log.
I20260928 06:32:37.530198 9631 log.cc:824] T 00000000000000000000000000000000 P dd32aba027bc4668a99a18cd74e41e79: Log is configured to *not* fsync() on all Append() calls
I20260928 06:32:37.530979 9631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd32aba027bc4668a99a18cd74e41e79: No bootstrap required, opened a new log
I20260928 06:32:37.533025 9631 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd32aba027bc4668a99a18cd74e41e79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd32aba027bc4668a99a18cd74e41e79" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 43159 } }
I20260928 06:32:37.533198 9631 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd32aba027bc4668a99a18cd74e41e79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:32:37.533279 9631 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd32aba027bc4668a99a18cd74e41e79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd32aba027bc4668a99a18cd74e41e79, State: Initialized, Role: FOLLOWER
I20260928 06:32:37.533481 9631 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd32aba027bc4668a99a18cd74e41e79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd32aba027bc4668a99a18cd74e41e79" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 43159 } }
I20260928 06:32:37.533592 9631 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd32aba027bc4668a99a18cd74e41e79 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:32:37.533632 9631 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd32aba027bc4668a99a18cd74e41e79 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:32:37.533653 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 9564
I20260928 06:32:37.533686 9631 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd32aba027bc4668a99a18cd74e41e79 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:32:37.533766 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576899143097-23430-0/minicluster-data/master-0/wal/instance
I20260928 06:32:37.533996 23430 external_mini_cluster.cc:1428] Setting key 03a9cf1d030659479845447cb5719169
I20260928 06:32:37.534480 9631 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd32aba027bc4668a99a18cd74e41e79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd32aba027bc4668a99a18cd74e41e79" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 43159 } }
I20260928 06:32:37.534652 9631 leader_election.cc:302] T 00000000000000000000000000000000 P dd32aba027bc4668a99a18cd74e41e79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd32aba027bc4668a99a18cd74e41e79; no voters:
I20260928 06:32:37.534862 9631 leader_election.cc:288] T 00000000000000000000000000000000 P dd32aba027bc4668a99a18cd74e41e79 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:32:37.534941 9636 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd32aba027bc4668a99a18cd74e41e79 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:32:37.535117 9636 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd32aba027bc4668a99a18cd74e41e79 [term 1 LEADER]: Becoming Leader. State: Replica: dd32aba027bc4668a99a18cd74e41e79, State: Running, Role: LEADER
I20260928 06:32:37.535157 9631 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd32aba027bc4668a99a18cd74e41e79 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:32:37.535454 9636 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd32aba027bc4668a99a18cd74e41e79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd32aba027bc4668a99a18cd74e41e79" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 43159 } }
I20260928 06:32:37.535859 9636 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd32aba027bc4668a99a18cd74e41e79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd32aba027bc4668a99a18cd74e41e79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd32aba027bc4668a99a18cd74e41e79" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 43159 } } }
I20260928 06:32:37.535987 9636 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd32aba027bc4668a99a18cd74e41e79 [sys.catalog]: This master's current role is: LEADER
I20260928 06:32:37.536146 9636 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd32aba027bc4668a99a18cd74e41e79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd32aba027bc4668a99a18cd74e41e79. Latest consensus state: current_term: 1 leader_uuid: "dd32aba027bc4668a99a18cd74e41e79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd32aba027bc4668a99a18cd74e41e79" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 43159 } } }
I20260928 06:32:37.536242 9636 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd32aba027bc4668a99a18cd74e41e79 [sys.catalog]: This master's current role is: LEADER
I20260928 06:32:37.537617 9645 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:32:37.538165 9645 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:32:37.539772 9645 catalog_manager.cc:1347] Generated new cluster ID: b6062efb7f8f4871820425ac91b51c42
I20260928 06:32:37.540205 9645 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:32:37.541198 9635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:37.534660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44260 (local address 127.22.225.190:43159)
0928 06:32:37.534909 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:37.534920 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:37.534957 (+ 37us) server_negotiation.cc:407] Connection header received
0928 06:32:37.535084 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:37.535091 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:37.535161 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:37.535568 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:32:37.535663 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:37.537476 (+ 1813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:37.538581 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:37.539200 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:37.539330 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:37.540312 (+ 982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:32:37.540316 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:32:37.540316 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:32:37.540533 (+ 217us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:32:37.540609 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:32:37.540628 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:32:37.540691 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:37.540899 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 06:32:37.540983 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":122,"threads_started":1}
I20260928 06:32:37.551903 9645 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:32:37.552766 9645 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:32:37.557351 9645 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd32aba027bc4668a99a18cd74e41e79: Generated new TSK 0
I20260928 06:32:37.557545 9645 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:32:37.562379 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576899143097-23430-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576899143097-23430-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576899143097-23430-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.129:0
--local_ip_for_outbound_sockets=127.22.225.129
--webserver_interface=127.22.225.129
--webserver_port=0
--tserver_master_addrs=127.22.225.190:43159
--builtin_ntp_servers=127.22.225.148:46879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 06:32:37.675448 9655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:32:37.675716 9655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:32:37.675760 9655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:32:37.677349 9655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:32:37.677428 9655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:32:37.677474 9655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.129
I20260928 06:32:37.679208 9655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:46879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576899143097-23430-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576899143097-23430-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.225.129
--webserver_port=0
--tserver_master_addrs=127.22.225.190:43159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.129
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576899143097-23430-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:32:37.679508 9655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:32:37.679836 9655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:32:37.685241 9661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:37.685562 9663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:37.685794 9660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:32:37.685982 9655 server_base.cc:1034] running on GCE node
I20260928 06:32:37.686203 9655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:32:37.686468 9655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:32:37.687638 9655 hybrid_clock.cc:648] HybridClock initialized: now 1790577157687617 us; error 43 us; skew 500 ppm
I20260928 06:32:37.689199 9655 webserver.cc:460] Webserver started at http://127.22.225.129:44635/ using document root <none> and password file <none>
I20260928 06:32:37.689534 9655 fs_manager.cc:359] Metadata directory not provided
I20260928 06:32:37.689618 9655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:32:37.689761 9655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:32:37.690871 9655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576899143097-23430-0/minicluster-data/ts-0/data/instance:
uuid: "069ce099ceb840c4992d338fd26f0614"
format_stamp: "Formatted at 2026-09-28 06:32:37 on dist-test-slave-d59m"
server_key: "b0ebf169d89528ef17dd4ac7483b2599"
server_key_iv: "53630d605958fccd3228a9f7b62142f3"
server_key_version: "encryptionkey@0"
I20260928 06:32:37.691170 9655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576899143097-23430-0/minicluster-data/ts-0/wal/instance:
uuid: "069ce099ceb840c4992d338fd26f0614"
format_stamp: "Formatted at 2026-09-28 06:32:37 on dist-test-slave-d59m"
server_key: "b0ebf169d89528ef17dd4ac7483b2599"
server_key_iv: "53630d605958fccd3228a9f7b62142f3"
server_key_version: "encryptionkey@0"
I20260928 06:32:37.692759 9655 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 06:32:37.693591 9669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:37.693771 9655 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:32:37.693859 9655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576899143097-23430-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576899143097-23430-0/minicluster-data/ts-0/wal
uuid: "069ce099ceb840c4992d338fd26f0614"
format_stamp: "Formatted at 2026-09-28 06:32:37 on dist-test-slave-d59m"
server_key: "b0ebf169d89528ef17dd4ac7483b2599"
server_key_iv: "53630d605958fccd3228a9f7b62142f3"
server_key_version: "encryptionkey@0"
I20260928 06:32:37.693958 9655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576899143097-23430-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576899143097-23430-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576899143097-23430-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:32:37.725235 9655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:32:37.725584 9655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:32:37.725751 9655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:32:37.726048 9655 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:32:37.726505 9655 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:32:37.726567 9655 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:37.726610 9655 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:32:37.726639 9655 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:37.733493 9655 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.129:35317
I20260928 06:32:37.733536 9782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.129:35317 every 8 connection(s)
I20260928 06:32:37.733876 9655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
I20260928 06:32:37.739841 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 9655
I20260928 06:32:37.739972 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576899143097-23430-0/minicluster-data/ts-0/wal/instance
I20260928 06:32:37.740206 23430 external_mini_cluster.cc:1428] Setting key 9ac1db43f2bf02c53df760ed62110fb3
I20260928 06:32:37.741122 9635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:37.734697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:52753 (local address 127.22.225.190:43159)
0928 06:32:37.734780 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:37.734782 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:37.734995 (+ 213us) server_negotiation.cc:407] Connection header received
0928 06:32:37.735557 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:37.735559 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:37.735574 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:37.735639 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:32:37.736792 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:37.737595 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:37.738795 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:37.739108 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:37.739178 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:37.740697 (+ 1519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:32:37.740699 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:32:37.740699 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:32:37.740704 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:32:37.740720 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:32:37.740725 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:32:37.740800 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:37.741011 (+ 211us) server_negotiation.cc:299] Negotiation successful
0928 06:32:37.741043 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 06:32:37.741271 9785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:37.734653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:43159 (local address 127.22.225.129:52753)
0928 06:32:37.734856 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:37.734900 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 06:32:37.735441 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:32:37.735739 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:32:37.735743 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:32:37.735863 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:32:37.736674 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:37.736683 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:37.737707 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:37.737708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:37.738699 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:37.738702 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:37.739224 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:37.739224 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:37.739339 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:37.740505 (+ 1166us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:32:37.740505 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:32:37.740547 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:32:37.740572 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:32:37.740802 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:32:37.740809 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:32:37.740813 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:32:37.740892 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 06:32:37.741018 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":92,"threads_started":1}
I20260928 06:32:37.741325 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576899143097-23430-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576899143097-23430-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576899143097-23430-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.130:0
--local_ip_for_outbound_sockets=127.22.225.130
--webserver_interface=127.22.225.130
--webserver_port=0
--tserver_master_addrs=127.22.225.190:43159
--builtin_ntp_servers=127.22.225.148:46879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 06:32:37.743007 9783 heartbeater.cc:347] Connected to a master server at 127.22.225.190:43159
I20260928 06:32:37.743147 9783 heartbeater.cc:464] Registering TS with master...
I20260928 06:32:37.743397 9783 heartbeater.cc:511] Master 127.22.225.190:43159 requested a full tablet report, sending...
I20260928 06:32:37.744023 9595 ts_manager.cc:194] Registered new tserver with Master: 069ce099ceb840c4992d338fd26f0614 (127.22.225.129:35317)
I20260928 06:32:37.744982 9595 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.225.129:52753
W20260928 06:32:37.857244 9786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:32:37.857502 9786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:32:37.857532 9786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:32:37.859143 9786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:32:37.859193 9786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:32:37.859237 9786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.130
I20260928 06:32:37.861218 9786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:46879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576899143097-23430-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576899143097-23430-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.225.130
--webserver_port=0
--tserver_master_addrs=127.22.225.190:43159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.130
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576899143097-23430-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:32:37.861479 9786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:32:37.861819 9786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:32:37.867522 9791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:37.867523 9794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:37.867517 9792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:32:37.867895 9786 server_base.cc:1034] running on GCE node
I20260928 06:32:37.868105 9786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:32:37.868367 9786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:32:37.869534 9786 hybrid_clock.cc:648] HybridClock initialized: now 1790577157869521 us; error 30 us; skew 500 ppm
I20260928 06:32:37.870939 9786 webserver.cc:460] Webserver started at http://127.22.225.130:43581/ using document root <none> and password file <none>
I20260928 06:32:37.871224 9786 fs_manager.cc:359] Metadata directory not provided
I20260928 06:32:37.871282 9786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:32:37.871409 9786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:32:37.872787 9786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576899143097-23430-0/minicluster-data/ts-1/data/instance:
uuid: "34cfb9766a4649129ad3dfc00dfa0323"
format_stamp: "Formatted at 2026-09-28 06:32:37 on dist-test-slave-d59m"
server_key: "558bbe552a520bc002202ddbe3cc61c3"
server_key_iv: "392daab8a2cca3ef198359fb2353e006"
server_key_version: "encryptionkey@0"
I20260928 06:32:37.873180 9786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576899143097-23430-0/minicluster-data/ts-1/wal/instance:
uuid: "34cfb9766a4649129ad3dfc00dfa0323"
format_stamp: "Formatted at 2026-09-28 06:32:37 on dist-test-slave-d59m"
server_key: "558bbe552a520bc002202ddbe3cc61c3"
server_key_iv: "392daab8a2cca3ef198359fb2353e006"
server_key_version: "encryptionkey@0"
I20260928 06:32:37.874828 9786 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:32:37.875680 9800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:37.875898 9786 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:32:37.875994 9786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576899143097-23430-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576899143097-23430-0/minicluster-data/ts-1/wal
uuid: "34cfb9766a4649129ad3dfc00dfa0323"
format_stamp: "Formatted at 2026-09-28 06:32:37 on dist-test-slave-d59m"
server_key: "558bbe552a520bc002202ddbe3cc61c3"
server_key_iv: "392daab8a2cca3ef198359fb2353e006"
server_key_version: "encryptionkey@0"
I20260928 06:32:37.876060 9786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576899143097-23430-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576899143097-23430-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576899143097-23430-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:32:37.905941 9786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:32:37.906306 9786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:32:37.906438 9786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:32:37.906684 9786 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:32:37.907037 9786 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:32:37.907080 9786 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:37.907110 9786 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:32:37.907126 9786 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:37.914041 9786 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.130:43833
I20260928 06:32:37.914135 9913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.130:43833 every 8 connection(s)
I20260928 06:32:37.914554 9786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
I20260928 06:32:37.917351 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 9786
I20260928 06:32:37.917466 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576899143097-23430-0/minicluster-data/ts-1/wal/instance
I20260928 06:32:37.917675 23430 external_mini_cluster.cc:1428] Setting key 7fa1947f007821ea280a07f1c9e64be9
I20260928 06:32:37.918619 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576899143097-23430-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576899143097-23430-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576899143097-23430-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.131:0
--local_ip_for_outbound_sockets=127.22.225.131
--webserver_interface=127.22.225.131
--webserver_port=0
--tserver_master_addrs=127.22.225.190:43159
--builtin_ntp_servers=127.22.225.148:46879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 06:32:37.921408 9635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:37.915337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:57231 (local address 127.22.225.190:43159)
0928 06:32:37.915404 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:37.915406 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:37.915641 (+ 235us) server_negotiation.cc:407] Connection header received
0928 06:32:37.916202 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:37.916204 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:37.916218 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:37.916278 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:32:37.917249 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:37.918039 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:37.919521 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:37.919862 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:37.919943 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:37.921026 (+ 1083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:32:37.921027 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:32:37.921028 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:32:37.921035 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:32:37.921054 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:32:37.921060 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:32:37.921122 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:37.921251 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 06:32:37.921284 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 06:32:37.921595 9916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:37.915321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:43159 (local address 127.22.225.130:57231)
0928 06:32:37.915511 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:37.915547 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 06:32:37.916077 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:32:37.916393 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:32:37.916398 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:32:37.916409 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:32:37.917116 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:37.917122 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:37.918155 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:37.918156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:37.919427 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:37.919431 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:37.920030 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:37.920032 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:37.920197 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:37.920858 (+ 661us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:32:37.920858 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:32:37.920889 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:32:37.920913 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:32:37.921143 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:32:37.921150 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:32:37.921153 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:32:37.921249 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 06:32:37.921363 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":86,"threads_started":1}
I20260928 06:32:37.922129 9914 heartbeater.cc:347] Connected to a master server at 127.22.225.190:43159
I20260928 06:32:37.922237 9914 heartbeater.cc:464] Registering TS with master...
I20260928 06:32:37.922441 9914 heartbeater.cc:511] Master 127.22.225.190:43159 requested a full tablet report, sending...
I20260928 06:32:37.922847 9595 ts_manager.cc:194] Registered new tserver with Master: 34cfb9766a4649129ad3dfc00dfa0323 (127.22.225.130:43833)
I20260928 06:32:37.923357 9595 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.225.130:57231
W20260928 06:32:38.007364 9917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:32:38.007598 9917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:32:38.007637 9917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:32:38.008848 9917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:32:38.008905 9917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:32:38.008947 9917 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.131
I20260928 06:32:38.010298 9917 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:46879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576899143097-23430-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576899143097-23430-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.225.131
--webserver_port=0
--tserver_master_addrs=127.22.225.190:43159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9917
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.131
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576899143097-23430-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:32:38.010558 9917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:32:38.010860 9917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:32:38.015980 9917 server_base.cc:1034] running on GCE node
W20260928 06:32:38.015904 9923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:38.016083 9922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:38.015900 9925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:32:38.016450 9917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:32:38.016744 9917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:32:38.017908 9917 hybrid_clock.cc:648] HybridClock initialized: now 1790577158017889 us; error 37 us; skew 500 ppm
I20260928 06:32:38.019483 9917 webserver.cc:460] Webserver started at http://127.22.225.131:40193/ using document root <none> and password file <none>
I20260928 06:32:38.019768 9917 fs_manager.cc:359] Metadata directory not provided
I20260928 06:32:38.019840 9917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:32:38.019979 9917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:32:38.021235 9917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576899143097-23430-0/minicluster-data/ts-2/data/instance:
uuid: "5d750e67b6234198aaf9bf8dcdf474b6"
format_stamp: "Formatted at 2026-09-28 06:32:38 on dist-test-slave-d59m"
server_key: "cd4d6fc8a65be145cb9fc69148af2625"
server_key_iv: "d0887b634ab97cb45b1cbbb088f08b12"
server_key_version: "encryptionkey@0"
I20260928 06:32:38.021638 9917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576899143097-23430-0/minicluster-data/ts-2/wal/instance:
uuid: "5d750e67b6234198aaf9bf8dcdf474b6"
format_stamp: "Formatted at 2026-09-28 06:32:38 on dist-test-slave-d59m"
server_key: "cd4d6fc8a65be145cb9fc69148af2625"
server_key_iv: "d0887b634ab97cb45b1cbbb088f08b12"
server_key_version: "encryptionkey@0"
I20260928 06:32:38.023037 9917 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:32:38.023818 9931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:38.023998 9917 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:32:38.024099 9917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576899143097-23430-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576899143097-23430-0/minicluster-data/ts-2/wal
uuid: "5d750e67b6234198aaf9bf8dcdf474b6"
format_stamp: "Formatted at 2026-09-28 06:32:38 on dist-test-slave-d59m"
server_key: "cd4d6fc8a65be145cb9fc69148af2625"
server_key_iv: "d0887b634ab97cb45b1cbbb088f08b12"
server_key_version: "encryptionkey@0"
I20260928 06:32:38.024194 9917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576899143097-23430-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576899143097-23430-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576899143097-23430-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:32:38.035085 9917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:32:38.035507 9917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:32:38.035674 9917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:32:38.035930 9917 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:32:38.036275 9917 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:32:38.036330 9917 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:38.036366 9917 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:32:38.036413 9917 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:38.044114 9917 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.131:39839
I20260928 06:32:38.044167 10044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.131:39839 every 8 connection(s)
I20260928 06:32:38.044687 9917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
I20260928 06:32:38.050446 9635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:38.045452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:40967 (local address 127.22.225.190:43159)
0928 06:32:38.045517 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:38.045519 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:38.045749 (+ 230us) server_negotiation.cc:407] Connection header received
0928 06:32:38.046218 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:38.046220 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:38.046231 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:38.046312 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:32:38.046963 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:38.047753 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:38.048809 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:38.049166 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:38.049245 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:38.050136 (+ 891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:32:38.050138 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:32:38.050138 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:32:38.050154 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:32:38.050169 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:32:38.050174 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:32:38.050247 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:38.050332 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 06:32:38.050364 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 06:32:38.050676 10047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:38.045432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:43159 (local address 127.22.225.131:40967)
0928 06:32:38.045610 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:38.045651 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 06:32:38.046126 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:32:38.046366 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:32:38.046370 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:32:38.046379 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:32:38.046879 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:38.046882 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:38.047845 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:38.047846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:38.048719 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:38.048722 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:38.049265 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:38.049265 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:38.049373 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:38.049970 (+ 597us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:32:38.049971 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:32:38.049999 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:32:38.050023 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:32:38.050251 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:32:38.050256 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:32:38.050260 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:32:38.050334 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 06:32:38.050439 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":80,"threads_started":1}
I20260928 06:32:38.051172 10045 heartbeater.cc:347] Connected to a master server at 127.22.225.190:43159
I20260928 06:32:38.051281 10045 heartbeater.cc:464] Registering TS with master...
I20260928 06:32:38.051473 10045 heartbeater.cc:511] Master 127.22.225.190:43159 requested a full tablet report, sending...
I20260928 06:32:38.051847 9595 ts_manager.cc:194] Registered new tserver with Master: 5d750e67b6234198aaf9bf8dcdf474b6 (127.22.225.131:39839)
I20260928 06:32:38.052328 9595 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.225.131:40967
I20260928 06:32:38.054448 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 9917
I20260928 06:32:38.054544 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576899143097-23430-0/minicluster-data/ts-2/wal/instance
I20260928 06:32:38.054738 23430 external_mini_cluster.cc:1428] Setting key e76745e28c71cb6fe1b5ecbb62850c0f
I20260928 06:32:38.055605 23430 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:32:38.060564 9635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:38.057044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44272 (local address 127.22.225.190:43159)
0928 06:32:38.057102 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:38.057104 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:38.057346 (+ 242us) server_negotiation.cc:407] Connection header received
0928 06:32:38.057501 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:38.057503 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:38.057519 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:38.057596 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:32:38.057824 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:38.058277 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:38.059331 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:38.059513 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:38.059567 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:38.060274 (+ 707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:32:38.060276 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:32:38.060276 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:32:38.060281 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:32:38.060295 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:32:38.060299 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:32:38.060403 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:38.060471 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 06:32:38.060481 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 06:32:38.062285 9595 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44272:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:32:38.064429 9595 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:32:38.071300 10059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:38.066736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.130:43833 (local address 127.0.0.1:57636)
0928 06:32:38.067055 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:38.067098 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:32:38.067168 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:32:38.067763 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:32:38.067764 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:32:38.067766 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:32:38.067809 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:38.067810 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:38.069110 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:38.069111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:38.070103 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:38.070105 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:38.071059 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:38.071060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:38.071120 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:38.071123 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:32:38.071169 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 06:32:38.071211 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":57,"threads_started":1}
I20260928 06:32:38.071332 10058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:38.066861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35266 (local address 127.22.225.129:35317)
0928 06:32:38.067057 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:38.067062 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:38.067252 (+ 190us) server_negotiation.cc:407] Connection header received
0928 06:32:38.067295 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:38.067297 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:38.067319 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:38.067406 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:32:38.067580 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:38.068379 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:38.069995 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:38.070753 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:38.070829 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:38.070921 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:38.071216 (+ 295us) server_negotiation.cc:299] Negotiation successful
0928 06:32:38.071256 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":87,"threads_started":1}
I20260928 06:32:38.071554 10057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:38.066739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.129:35317 (local address 127.0.0.1:35266)
0928 06:32:38.067103 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:38.067112 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:32:38.067167 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:32:38.067408 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:32:38.067410 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:32:38.067413 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:32:38.067487 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:38.067489 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:38.068472 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:38.068473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:38.069911 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:38.069913 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:38.070959 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:38.070960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:38.071041 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:38.071056 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 06:32:38.071121 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 06:32:38.071500 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":80,"threads_started":1}
I20260928 06:32:38.071913 10061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:38.067310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57636 (local address 127.22.225.130:43833)
0928 06:32:38.067542 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:38.067546 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:38.067568 (+ 22us) server_negotiation.cc:407] Connection header received
0928 06:32:38.067606 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:38.067608 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:38.067622 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:38.067700 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:32:38.067876 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:38.069023 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:38.070208 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:38.070892 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:38.070954 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:38.071028 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:38.071770 (+ 742us) server_negotiation.cc:299] Negotiation successful
0928 06:32:38.071818 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":76,"threads_started":1}
I20260928 06:32:38.072773 9848 tablet_service.cc:1467] Processing CreateTablet for tablet 702282e2b1fa46aa85a537c2392071be (DEFAULT_TABLE table=test-table [id=42cc72dfb0e24d938f6ca873253b5ddc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:32:38.073148 9848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 702282e2b1fa46aa85a537c2392071be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:32:38.073136 9717 tablet_service.cc:1467] Processing CreateTablet for tablet 702282e2b1fa46aa85a537c2392071be (DEFAULT_TABLE table=test-table [id=42cc72dfb0e24d938f6ca873253b5ddc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:32:38.073431 9717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 702282e2b1fa46aa85a537c2392071be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:32:38.073593 10056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:38.066925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.131:39839 (local address 127.0.0.1:35034)
0928 06:32:38.069139 (+ 2214us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:38.069149 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:32:38.069194 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:32:38.069448 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:32:38.069448 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:32:38.069450 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:32:38.069498 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:38.069499 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:38.070796 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:38.070797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:38.072322 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:38.072325 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:38.073366 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:38.073367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:38.073427 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:38.073431 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:32:38.073475 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 06:32:38.073511 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2199,"thread_start_us":79,"threads_started":1}
I20260928 06:32:38.073843 10060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:38.067143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35034 (local address 127.22.225.131:39839)
0928 06:32:38.067481 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:38.067486 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:38.069262 (+ 1776us) server_negotiation.cc:407] Connection header received
0928 06:32:38.069308 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:38.069310 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:38.069326 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:38.069388 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:32:38.069563 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:38.070704 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:38.072411 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:38.073176 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:38.073255 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:38.073331 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:38.073726 (+ 395us) server_negotiation.cc:299] Negotiation successful
0928 06:32:38.073765 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":85,"threads_started":1}
I20260928 06:32:38.074385 9979 tablet_service.cc:1467] Processing CreateTablet for tablet 702282e2b1fa46aa85a537c2392071be (DEFAULT_TABLE table=test-table [id=42cc72dfb0e24d938f6ca873253b5ddc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:32:38.074666 9979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 702282e2b1fa46aa85a537c2392071be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:32:38.076550 10062 tablet_bootstrap.cc:492] T 702282e2b1fa46aa85a537c2392071be P 34cfb9766a4649129ad3dfc00dfa0323: Bootstrap starting.
I20260928 06:32:38.077241 10062 tablet_bootstrap.cc:654] T 702282e2b1fa46aa85a537c2392071be P 34cfb9766a4649129ad3dfc00dfa0323: Neither blocks nor log segments found. Creating new log.
I20260928 06:32:38.077574 10062 log.cc:824] T 702282e2b1fa46aa85a537c2392071be P 34cfb9766a4649129ad3dfc00dfa0323: Log is configured to *not* fsync() on all Append() calls
I20260928 06:32:38.077911 10063 tablet_bootstrap.cc:492] T 702282e2b1fa46aa85a537c2392071be P 069ce099ceb840c4992d338fd26f0614: Bootstrap starting.
I20260928 06:32:38.077922 10064 tablet_bootstrap.cc:492] T 702282e2b1fa46aa85a537c2392071be P 5d750e67b6234198aaf9bf8dcdf474b6: Bootstrap starting.
I20260928 06:32:38.078425 10062 tablet_bootstrap.cc:492] T 702282e2b1fa46aa85a537c2392071be P 34cfb9766a4649129ad3dfc00dfa0323: No bootstrap required, opened a new log
I20260928 06:32:38.078534 10062 ts_tablet_manager.cc:1397] T 702282e2b1fa46aa85a537c2392071be P 34cfb9766a4649129ad3dfc00dfa0323: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 06:32:38.078536 10063 tablet_bootstrap.cc:654] T 702282e2b1fa46aa85a537c2392071be P 069ce099ceb840c4992d338fd26f0614: Neither blocks nor log segments found. Creating new log.
I20260928 06:32:38.078612 10064 tablet_bootstrap.cc:654] T 702282e2b1fa46aa85a537c2392071be P 5d750e67b6234198aaf9bf8dcdf474b6: Neither blocks nor log segments found. Creating new log.
I20260928 06:32:38.078837 10063 log.cc:824] T 702282e2b1fa46aa85a537c2392071be P 069ce099ceb840c4992d338fd26f0614: Log is configured to *not* fsync() on all Append() calls
I20260928 06:32:38.078945 10064 log.cc:824] T 702282e2b1fa46aa85a537c2392071be P 5d750e67b6234198aaf9bf8dcdf474b6: Log is configured to *not* fsync() on all Append() calls
I20260928 06:32:38.079624 10064 tablet_bootstrap.cc:492] T 702282e2b1fa46aa85a537c2392071be P 5d750e67b6234198aaf9bf8dcdf474b6: No bootstrap required, opened a new log
I20260928 06:32:38.079665 10063 tablet_bootstrap.cc:492] T 702282e2b1fa46aa85a537c2392071be P 069ce099ceb840c4992d338fd26f0614: No bootstrap required, opened a new log
I20260928 06:32:38.079753 10063 ts_tablet_manager.cc:1397] T 702282e2b1fa46aa85a537c2392071be P 069ce099ceb840c4992d338fd26f0614: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:32:38.079769 10064 ts_tablet_manager.cc:1397] T 702282e2b1fa46aa85a537c2392071be P 5d750e67b6234198aaf9bf8dcdf474b6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:32:38.081549 10062 raft_consensus.cc:348] T 702282e2b1fa46aa85a537c2392071be P 34cfb9766a4649129ad3dfc00dfa0323 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34cfb9766a4649129ad3dfc00dfa0323" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 43833 } } peers { permanent_uuid: "069ce099ceb840c4992d338fd26f0614" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 35317 } } peers { permanent_uuid: "5d750e67b6234198aaf9bf8dcdf474b6" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 39839 } }
I20260928 06:32:38.081739 10062 raft_consensus.cc:374] T 702282e2b1fa46aa85a537c2392071be P 34cfb9766a4649129ad3dfc00dfa0323 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:32:38.081784 10062 raft_consensus.cc:729] T 702282e2b1fa46aa85a537c2392071be P 34cfb9766a4649129ad3dfc00dfa0323 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34cfb9766a4649129ad3dfc00dfa0323, State: Initialized, Role: FOLLOWER
I20260928 06:32:38.081895 10062 consensus_queue.cc:260] T 702282e2b1fa46aa85a537c2392071be P 34cfb9766a4649129ad3dfc00dfa0323 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34cfb9766a4649129ad3dfc00dfa0323" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 43833 } } peers { permanent_uuid: "069ce099ceb840c4992d338fd26f0614" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 35317 } } peers { permanent_uuid: "5d750e67b6234198aaf9bf8dcdf474b6" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 39839 } }
I20260928 06:32:38.082157 10062 ts_tablet_manager.cc:1428] T 702282e2b1fa46aa85a537c2392071be P 34cfb9766a4649129ad3dfc00dfa0323: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260928 06:32:38.082393 10063 raft_consensus.cc:348] T 702282e2b1fa46aa85a537c2392071be P 069ce099ceb840c4992d338fd26f0614 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34cfb9766a4649129ad3dfc00dfa0323" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 43833 } } peers { permanent_uuid: "069ce099ceb840c4992d338fd26f0614" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 35317 } } peers { permanent_uuid: "5d750e67b6234198aaf9bf8dcdf474b6" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 39839 } }
I20260928 06:32:38.082531 10063 raft_consensus.cc:374] T 702282e2b1fa46aa85a537c2392071be P 069ce099ceb840c4992d338fd26f0614 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:32:38.082572 10063 raft_consensus.cc:729] T 702282e2b1fa46aa85a537c2392071be P 069ce099ceb840c4992d338fd26f0614 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 069ce099ceb840c4992d338fd26f0614, State: Initialized, Role: FOLLOWER
I20260928 06:32:38.082675 10063 consensus_queue.cc:260] T 702282e2b1fa46aa85a537c2392071be P 069ce099ceb840c4992d338fd26f0614 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34cfb9766a4649129ad3dfc00dfa0323" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 43833 } } peers { permanent_uuid: "069ce099ceb840c4992d338fd26f0614" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 35317 } } peers { permanent_uuid: "5d750e67b6234198aaf9bf8dcdf474b6" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 39839 } }
I20260928 06:32:38.082773 10064 raft_consensus.cc:348] T 702282e2b1fa46aa85a537c2392071be P 5d750e67b6234198aaf9bf8dcdf474b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34cfb9766a4649129ad3dfc00dfa0323" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 43833 } } peers { permanent_uuid: "069ce099ceb840c4992d338fd26f0614" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 35317 } } peers { permanent_uuid: "5d750e67b6234198aaf9bf8dcdf474b6" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 39839 } }
I20260928 06:32:38.082921 10064 raft_consensus.cc:374] T 702282e2b1fa46aa85a537c2392071be P 5d750e67b6234198aaf9bf8dcdf474b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:32:38.082953 10064 raft_consensus.cc:729] T 702282e2b1fa46aa85a537c2392071be P 5d750e67b6234198aaf9bf8dcdf474b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d750e67b6234198aaf9bf8dcdf474b6, State: Initialized, Role: FOLLOWER
I20260928 06:32:38.082950 10063 ts_tablet_manager.cc:1428] T 702282e2b1fa46aa85a537c2392071be P 069ce099ceb840c4992d338fd26f0614: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 06:32:38.083087 10064 consensus_queue.cc:260] T 702282e2b1fa46aa85a537c2392071be P 5d750e67b6234198aaf9bf8dcdf474b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34cfb9766a4649129ad3dfc00dfa0323" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 43833 } } peers { permanent_uuid: "069ce099ceb840c4992d338fd26f0614" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 35317 } } peers { permanent_uuid: "5d750e67b6234198aaf9bf8dcdf474b6" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 39839 } }
I20260928 06:32:38.083266 9914 heartbeater.cc:503] Master 127.22.225.190:43159 was elected leader, sending a full tablet report...
I20260928 06:32:38.083304 10064 ts_tablet_manager.cc:1428] T 702282e2b1fa46aa85a537c2392071be P 5d750e67b6234198aaf9bf8dcdf474b6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:32:38.083413 10045 heartbeater.cc:503] Master 127.22.225.190:43159 was elected leader, sending a full tablet report...
I20260928 06:32:38.083786 9783 heartbeater.cc:503] Master 127.22.225.190:43159 was elected leader, sending a full tablet report...
W20260928 06:32:38.165098 9915 tablet.cc:2369] T 702282e2b1fa46aa85a537c2392071be P 34cfb9766a4649129ad3dfc00dfa0323: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:32:38.234669 9784 tablet.cc:2369] T 702282e2b1fa46aa85a537c2392071be P 069ce099ceb840c4992d338fd26f0614: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:32:38.295262 10046 tablet.cc:2369] T 702282e2b1fa46aa85a537c2392071be P 5d750e67b6234198aaf9bf8dcdf474b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:32:38.336668 10069 raft_consensus.cc:482] T 702282e2b1fa46aa85a537c2392071be P 069ce099ceb840c4992d338fd26f0614 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:32:38.336829 10069 raft_consensus.cc:504] T 702282e2b1fa46aa85a537c2392071be P 069ce099ceb840c4992d338fd26f0614 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34cfb9766a4649129ad3dfc00dfa0323" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 43833 } } peers { permanent_uuid: "069ce099ceb840c4992d338fd26f0614" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 35317 } } peers { permanent_uuid: "5d750e67b6234198aaf9bf8dcdf474b6" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 39839 } }
I20260928 06:32:38.337235 10069 leader_election.cc:288] T 702282e2b1fa46aa85a537c2392071be P 069ce099ceb840c4992d338fd26f0614 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 34cfb9766a4649129ad3dfc00dfa0323 (127.22.225.130:43833), 5d750e67b6234198aaf9bf8dcdf474b6 (127.22.225.131:39839)
I20260928 06:32:38.341037 10060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:38.337699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:49725 (local address 127.22.225.131:39839)
0928 06:32:38.337755 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:38.337758 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:38.337810 (+ 52us) server_negotiation.cc:407] Connection header received
0928 06:32:38.337924 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:38.337925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:38.337936 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:38.337983 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:32:38.338157 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:38.338830 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:38.340248 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:38.340652 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:38.340756 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:38.340766 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:38.340939 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 06:32:38.340977 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 06:32:38.341050 10072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:38.337629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.131:39839 (local address 127.22.225.129:49725)
0928 06:32:38.337753 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:38.337764 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:32:38.337828 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:32:38.338028 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:32:38.338029 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:32:38.338031 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:32:38.338102 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:38.338104 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:38.338916 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:38.338917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:38.340131 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:38.340133 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:38.340777 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:38.340779 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:38.340855 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:38.340859 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:32:38.340915 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 06:32:38.340957 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":56,"threads_started":1}
I20260928 06:32:38.341506 9999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "702282e2b1fa46aa85a537c2392071be" candidate_uuid: "069ce099ceb840c4992d338fd26f0614" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d750e67b6234198aaf9bf8dcdf474b6" is_pre_election: true
I20260928 06:32:38.341692 9999 raft_consensus.cc:2454] T 702282e2b1fa46aa85a537c2392071be P 5d750e67b6234198aaf9bf8dcdf474b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 069ce099ceb840c4992d338fd26f0614 in term 0.
I20260928 06:32:38.341961 9672 leader_election.cc:302] T 702282e2b1fa46aa85a537c2392071be P 069ce099ceb840c4992d338fd26f0614 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 069ce099ceb840c4992d338fd26f0614, 5d750e67b6234198aaf9bf8dcdf474b6; no voters:
I20260928 06:32:38.342163 10069 raft_consensus.cc:2781] T 702282e2b1fa46aa85a537c2392071be P 069ce099ceb840c4992d338fd26f0614 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:32:38.342219 10069 raft_consensus.cc:482] T 702282e2b1fa46aa85a537c2392071be P 069ce099ceb840c4992d338fd26f0614 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:32:38.342259 10069 raft_consensus.cc:3037] T 702282e2b1fa46aa85a537c2392071be P 069ce099ceb840c4992d338fd26f0614 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:32:38.342445 10071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:38.337448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.130:43833 (local address 127.22.225.129:32975)
0928 06:32:38.337677 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:38.337689 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:32:38.337785 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:32:38.337955 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:32:38.337956 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:32:38.337957 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:32:38.338028 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:38.338030 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:38.340256 (+ 2226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:38.340256 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:38.341453 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:38.341456 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:38.342209 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:38.342210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:38.342284 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:38.342286 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:32:38.342332 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 06:32:38.342367 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":79,"threads_started":1}
I20260928 06:32:38.342465 10061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:38.337535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:32975 (local address 127.22.225.130:43833)
0928 06:32:38.337600 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:38.337602 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:38.337757 (+ 155us) server_negotiation.cc:407] Connection header received
0928 06:32:38.337875 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:38.337877 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:38.337890 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:38.337951 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:32:38.338114 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:38.338840 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:38.341556 (+ 2716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:38.342098 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:38.342144 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:38.342151 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:38.342346 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 06:32:38.342395 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 06:32:38.343076 10069 raft_consensus.cc:504] T 702282e2b1fa46aa85a537c2392071be P 069ce099ceb840c4992d338fd26f0614 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34cfb9766a4649129ad3dfc00dfa0323" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 43833 } } peers { permanent_uuid: "069ce099ceb840c4992d338fd26f0614" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 35317 } } peers { permanent_uuid: "5d750e67b6234198aaf9bf8dcdf474b6" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 39839 } }
I20260928 06:32:38.343163 10069 leader_election.cc:288] T 702282e2b1fa46aa85a537c2392071be P 069ce099ceb840c4992d338fd26f0614 [CANDIDATE]: Term 1 election: Requested vote from peers 34cfb9766a4649129ad3dfc00dfa0323 (127.22.225.130:43833), 5d750e67b6234198aaf9bf8dcdf474b6 (127.22.225.131:39839)
I20260928 06:32:38.343571 9999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "702282e2b1fa46aa85a537c2392071be" candidate_uuid: "069ce099ceb840c4992d338fd26f0614" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d750e67b6234198aaf9bf8dcdf474b6"
I20260928 06:32:38.343672 9868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "702282e2b1fa46aa85a537c2392071be" candidate_uuid: "069ce099ceb840c4992d338fd26f0614" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34cfb9766a4649129ad3dfc00dfa0323" is_pre_election: true
I20260928 06:32:38.343668 9999 raft_consensus.cc:3037] T 702282e2b1fa46aa85a537c2392071be P 5d750e67b6234198aaf9bf8dcdf474b6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:32:38.343812 9868 raft_consensus.cc:2454] T 702282e2b1fa46aa85a537c2392071be P 34cfb9766a4649129ad3dfc00dfa0323 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 069ce099ceb840c4992d338fd26f0614 in term 0.
I20260928 06:32:38.343964 9867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "702282e2b1fa46aa85a537c2392071be" candidate_uuid: "069ce099ceb840c4992d338fd26f0614" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34cfb9766a4649129ad3dfc00dfa0323"
I20260928 06:32:38.344033 9867 raft_consensus.cc:3037] T 702282e2b1fa46aa85a537c2392071be P 34cfb9766a4649129ad3dfc00dfa0323 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:32:38.344662 9999 raft_consensus.cc:2454] T 702282e2b1fa46aa85a537c2392071be P 5d750e67b6234198aaf9bf8dcdf474b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 069ce099ceb840c4992d338fd26f0614 in term 1.
I20260928 06:32:38.344933 9867 raft_consensus.cc:2454] T 702282e2b1fa46aa85a537c2392071be P 34cfb9766a4649129ad3dfc00dfa0323 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 069ce099ceb840c4992d338fd26f0614 in term 1.
I20260928 06:32:38.345158 9672 leader_election.cc:302] T 702282e2b1fa46aa85a537c2392071be P 069ce099ceb840c4992d338fd26f0614 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 069ce099ceb840c4992d338fd26f0614, 5d750e67b6234198aaf9bf8dcdf474b6; no voters:
I20260928 06:32:38.345249 10069 raft_consensus.cc:2781] T 702282e2b1fa46aa85a537c2392071be P 069ce099ceb840c4992d338fd26f0614 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:32:38.345463 10069 raft_consensus.cc:686] T 702282e2b1fa46aa85a537c2392071be P 069ce099ceb840c4992d338fd26f0614 [term 1 LEADER]: Becoming Leader. State: Replica: 069ce099ceb840c4992d338fd26f0614, State: Running, Role: LEADER
I20260928 06:32:38.345558 10069 consensus_queue.cc:237] T 702282e2b1fa46aa85a537c2392071be P 069ce099ceb840c4992d338fd26f0614 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34cfb9766a4649129ad3dfc00dfa0323" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 43833 } } peers { permanent_uuid: "069ce099ceb840c4992d338fd26f0614" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 35317 } } peers { permanent_uuid: "5d750e67b6234198aaf9bf8dcdf474b6" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 39839 } }
I20260928 06:32:38.346424 9595 catalog_manager.cc:5495] T 702282e2b1fa46aa85a537c2392071be P 069ce099ceb840c4992d338fd26f0614 reported cstate change: term changed from 0 to 1, leader changed from <none> to 069ce099ceb840c4992d338fd26f0614 (127.22.225.129). New cstate: current_term: 1 leader_uuid: "069ce099ceb840c4992d338fd26f0614" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34cfb9766a4649129ad3dfc00dfa0323" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 43833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "069ce099ceb840c4992d338fd26f0614" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 35317 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d750e67b6234198aaf9bf8dcdf474b6" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 39839 } health_report { overall_health: UNKNOWN } } }
I20260928 06:32:38.359062 10058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:38.356425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35274 (local address 127.22.225.129:35317)
0928 06:32:38.356474 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:38.356476 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:38.356485 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:32:38.356526 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:38.356527 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:38.356549 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:38.356678 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:32:38.356843 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:38.357341 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:38.358304 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:38.358511 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:38.358586 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:38.358734 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:32:38.358830 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:32:38.358940 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:38.358970 (+ 30us) server_negotiation.cc:299] Negotiation successful
0928 06:32:38.358980 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 06:32:38.361052 9999 raft_consensus.cc:1261] T 702282e2b1fa46aa85a537c2392071be P 5d750e67b6234198aaf9bf8dcdf474b6 [term 1 FOLLOWER]: Refusing update from remote peer 069ce099ceb840c4992d338fd26f0614: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:32:38.361058 9867 raft_consensus.cc:1261] T 702282e2b1fa46aa85a537c2392071be P 34cfb9766a4649129ad3dfc00dfa0323 [term 1 FOLLOWER]: Refusing update from remote peer 069ce099ceb840c4992d338fd26f0614: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:32:38.361451 10073 consensus_queue.cc:1035] T 702282e2b1fa46aa85a537c2392071be P 069ce099ceb840c4992d338fd26f0614 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d750e67b6234198aaf9bf8dcdf474b6" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 39839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:32:38.361570 10069 consensus_queue.cc:1035] T 702282e2b1fa46aa85a537c2392071be P 069ce099ceb840c4992d338fd26f0614 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34cfb9766a4649129ad3dfc00dfa0323" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 43833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:32:38.364413 10077 mvcc.cc:205] Tried to move back new op lower bound from 7334204040643862528 to 7334204040583991296. Current Snapshot: MvccSnapshot[applied={T|T < 7334204040643862528 or (T in {7334204040643862528})}]
I20260928 06:32:38.364622 10078 mvcc.cc:205] Tried to move back new op lower bound from 7334204040643862528 to 7334204040583991296. Current Snapshot: MvccSnapshot[applied={T|T < 7334204040643862528}]
I20260928 06:32:38.365461 10076 mvcc.cc:205] Tried to move back new op lower bound from 7334204040643862528 to 7334204040583991296. Current Snapshot: MvccSnapshot[applied={T|T < 7334204040643862528 or (T in {7334204040643862528})}]
I20260928 06:32:38.366959 9594 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:32:38.367053 9594 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:32:38.367833 9594 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dd32aba027bc4668a99a18cd74e41e79: Sending DeleteTablet for 3 replicas of tablet 702282e2b1fa46aa85a537c2392071be
I20260928 06:32:38.368214 9717 tablet_service.cc:1514] Processing DeleteTablet for tablet 702282e2b1fa46aa85a537c2392071be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:32:38 UTC) from {username='slave'} at 127.0.0.1:35266
I20260928 06:32:38.368462 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 9655
I20260928 06:32:38.368685 9848 tablet_service.cc:1514] Processing DeleteTablet for tablet 702282e2b1fa46aa85a537c2392071be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:32:38 UTC) from {username='slave'} at 127.0.0.1:57636
I20260928 06:32:38.368872 9979 tablet_service.cc:1514] Processing DeleteTablet for tablet 702282e2b1fa46aa85a537c2392071be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:32:38 UTC) from {username='slave'} at 127.0.0.1:35034
I20260928 06:32:38.369215 10087 tablet_replica.cc:330] T 702282e2b1fa46aa85a537c2392071be P 5d750e67b6234198aaf9bf8dcdf474b6: stopping tablet replica
I20260928 06:32:38.369328 10087 raft_consensus.cc:2229] T 702282e2b1fa46aa85a537c2392071be P 5d750e67b6234198aaf9bf8dcdf474b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:32:38.369418 10087 raft_consensus.cc:2258] T 702282e2b1fa46aa85a537c2392071be P 5d750e67b6234198aaf9bf8dcdf474b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:32:38.369590 10086 tablet_replica.cc:330] T 702282e2b1fa46aa85a537c2392071be P 34cfb9766a4649129ad3dfc00dfa0323: stopping tablet replica
I20260928 06:32:38.369685 10086 raft_consensus.cc:2229] T 702282e2b1fa46aa85a537c2392071be P 34cfb9766a4649129ad3dfc00dfa0323 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:32:38.369767 10086 raft_consensus.cc:2258] T 702282e2b1fa46aa85a537c2392071be P 34cfb9766a4649129ad3dfc00dfa0323 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:32:38.369871 10087 ts_tablet_manager.cc:1905] T 702282e2b1fa46aa85a537c2392071be P 5d750e67b6234198aaf9bf8dcdf474b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:32:38.370196 10086 ts_tablet_manager.cc:1905] T 702282e2b1fa46aa85a537c2392071be P 34cfb9766a4649129ad3dfc00dfa0323: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:32:38.371838 10086 ts_tablet_manager.cc:1918] T 702282e2b1fa46aa85a537c2392071be P 34cfb9766a4649129ad3dfc00dfa0323: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:32:38.371901 10086 log.cc:1196] T 702282e2b1fa46aa85a537c2392071be P 34cfb9766a4649129ad3dfc00dfa0323: Deleting WAL directory at /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576899143097-23430-0/minicluster-data/ts-1/wal/wals/702282e2b1fa46aa85a537c2392071be
I20260928 06:32:38.371908 10087 ts_tablet_manager.cc:1918] T 702282e2b1fa46aa85a537c2392071be P 5d750e67b6234198aaf9bf8dcdf474b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:32:38.371976 10087 log.cc:1196] T 702282e2b1fa46aa85a537c2392071be P 5d750e67b6234198aaf9bf8dcdf474b6: Deleting WAL directory at /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576899143097-23430-0/minicluster-data/ts-2/wal/wals/702282e2b1fa46aa85a537c2392071be
I20260928 06:32:38.372136 10086 ts_tablet_manager.cc:1939] T 702282e2b1fa46aa85a537c2392071be P 34cfb9766a4649129ad3dfc00dfa0323: Deleting consensus metadata
I20260928 06:32:38.372279 10087 ts_tablet_manager.cc:1939] T 702282e2b1fa46aa85a537c2392071be P 5d750e67b6234198aaf9bf8dcdf474b6: Deleting consensus metadata
I20260928 06:32:38.372548 9581 catalog_manager.cc:4841] TS 34cfb9766a4649129ad3dfc00dfa0323 (127.22.225.130:43833): tablet 702282e2b1fa46aa85a537c2392071be (table test-table [id=42cc72dfb0e24d938f6ca873253b5ddc]) successfully deleted
I20260928 06:32:38.372784 9581 catalog_manager.cc:4841] TS 5d750e67b6234198aaf9bf8dcdf474b6 (127.22.225.131:39839): tablet 702282e2b1fa46aa85a537c2392071be (table test-table [id=42cc72dfb0e24d938f6ca873253b5ddc]) successfully deleted
W20260928 06:32:38.376650 9580 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.225.129:35317 (error 108)
I20260928 06:32:38.376861 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 9786
I20260928 06:32:38.377540 10056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:38.377024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.129:35317 (local address 127.0.0.1:35276)
0928 06:32:38.377460 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:38.377499 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.129:35317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415}
W20260928 06:32:38.378126 9580 catalog_manager.cc:4584] TS 069ce099ceb840c4992d338fd26f0614 (127.22.225.129:35317): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 702282e2b1fa46aa85a537c2392071be: Network error: Client connection negotiation failed: client connection to 127.22.225.129:35317: connect: Connection refused (error 111)
I20260928 06:32:38.381985 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 9917
I20260928 06:32:38.387372 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 9564
2026-09-28T06:32:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1050 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:32:38 dist-test-slave-d59m krb5kdc[10091](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:32:38 dist-test-slave-d59m krb5kdc[10091](info): set up 2 sockets
Sep 28 06:32:38 dist-test-slave-d59m krb5kdc[10091](info): commencing operation
krb5kdc: starting...
W20260928 06:32:39.505061 23430 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Sep 28 06:32:39 dist-test-slave-d59m krb5kdc[10091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:32:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:32:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab.
Entry for principal kudu/127.22.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab.
I20260928 06:32:39.631702 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:41699
--webserver_interface=127.22.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab
--principal=kudu/127.22.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:41853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.225.190:41699
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:32:39.715158 10104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:32:39.715366 10104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:32:39.715387 10104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:32:39.716652 10104 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:32:39.716691 10104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:32:39.716706 10104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:32:39.716717 10104 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:32:39.716764 10104 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:32:39.718299 10104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:41853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.225.190:41699
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab
--principal=kudu/127.22.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:41699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:32:39.718575 10104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:32:39.718811 10104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:32:39.723850 10112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:39.723917 10110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:39.723878 10109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:32:39.724282 10104 server_base.cc:1034] running on GCE node
I20260928 06:32:39.724506 10104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:32:39.724891 10104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:32:39.726084 10104 hybrid_clock.cc:648] HybridClock initialized: now 1790577159726064 us; error 41 us; skew 500 ppm
Sep 28 06:32:39 dist-test-slave-d59m krb5kdc[10091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577159, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:32:39.728825 10104 init.cc:377] Logged in from keytab as kudu/127.22.225.190@KRBTEST.COM (short username kudu)
I20260928 06:32:39.729312 10104 webserver.cc:460] Webserver started at http://127.22.225.190:32833/ using document root <none> and password file <none>
I20260928 06:32:39.729576 10104 fs_manager.cc:359] Metadata directory not provided
I20260928 06:32:39.729645 10104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:32:39.729812 10104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:32:39.730919 10104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/minicluster-data/master-0/data/instance:
uuid: "6b94cab2c92e410b93d665327013821d"
format_stamp: "Formatted at 2026-09-28 06:32:39 on dist-test-slave-d59m"
server_key: "f8065615df11f88efffdfb21064e2817"
server_key_iv: "3df17b98c2bd919e4cacc4da5c022870"
server_key_version: "encryptionkey@0"
I20260928 06:32:39.731297 10104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/minicluster-data/master-0/wal/instance:
uuid: "6b94cab2c92e410b93d665327013821d"
format_stamp: "Formatted at 2026-09-28 06:32:39 on dist-test-slave-d59m"
server_key: "f8065615df11f88efffdfb21064e2817"
server_key_iv: "3df17b98c2bd919e4cacc4da5c022870"
server_key_version: "encryptionkey@0"
I20260928 06:32:39.732707 10104 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:32:39.733497 10119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:39.733726 10104 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:32:39.733837 10104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/minicluster-data/master-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/minicluster-data/master-0/wal
uuid: "6b94cab2c92e410b93d665327013821d"
format_stamp: "Formatted at 2026-09-28 06:32:39 on dist-test-slave-d59m"
server_key: "f8065615df11f88efffdfb21064e2817"
server_key_iv: "3df17b98c2bd919e4cacc4da5c022870"
server_key_version: "encryptionkey@0"
I20260928 06:32:39.733906 10104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:32:39.746667 10104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:32:39.749109 10104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:32:39.749249 10104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:32:39.753929 10104 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.190:41699
I20260928 06:32:39.753996 10171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.190:41699 every 8 connection(s)
I20260928 06:32:39.754367 10104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
I20260928 06:32:39.755442 10172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:32:39.756896 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 10104
I20260928 06:32:39.757009 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/minicluster-data/master-0/wal/instance
I20260928 06:32:39.757195 23430 external_mini_cluster.cc:1428] Setting key d22c7c3ff53bd2a4d5d7d10b2c64023d
I20260928 06:32:39.758453 10172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b94cab2c92e410b93d665327013821d: Bootstrap starting.
I20260928 06:32:39.759208 10172 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b94cab2c92e410b93d665327013821d: Neither blocks nor log segments found. Creating new log.
I20260928 06:32:39.759562 10172 log.cc:824] T 00000000000000000000000000000000 P 6b94cab2c92e410b93d665327013821d: Log is configured to *not* fsync() on all Append() calls
I20260928 06:32:39.760389 10172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b94cab2c92e410b93d665327013821d: No bootstrap required, opened a new log
I20260928 06:32:39.762634 10172 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b94cab2c92e410b93d665327013821d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b94cab2c92e410b93d665327013821d" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 41699 } }
I20260928 06:32:39.762797 10172 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b94cab2c92e410b93d665327013821d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:32:39.762843 10172 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b94cab2c92e410b93d665327013821d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b94cab2c92e410b93d665327013821d, State: Initialized, Role: FOLLOWER
I20260928 06:32:39.762969 10172 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b94cab2c92e410b93d665327013821d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b94cab2c92e410b93d665327013821d" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 41699 } }
I20260928 06:32:39.763068 10172 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b94cab2c92e410b93d665327013821d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:32:39.763108 10172 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b94cab2c92e410b93d665327013821d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:32:39.763157 10172 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b94cab2c92e410b93d665327013821d [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:32:39.763841 10172 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b94cab2c92e410b93d665327013821d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b94cab2c92e410b93d665327013821d" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 41699 } }
I20260928 06:32:39.763963 10172 leader_election.cc:302] T 00000000000000000000000000000000 P 6b94cab2c92e410b93d665327013821d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b94cab2c92e410b93d665327013821d; no voters:
I20260928 06:32:39.764144 10172 leader_election.cc:288] T 00000000000000000000000000000000 P 6b94cab2c92e410b93d665327013821d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:32:39.764228 10177 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b94cab2c92e410b93d665327013821d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:32:39.764395 10177 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b94cab2c92e410b93d665327013821d [term 1 LEADER]: Becoming Leader. State: Replica: 6b94cab2c92e410b93d665327013821d, State: Running, Role: LEADER
I20260928 06:32:39.764436 10172 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b94cab2c92e410b93d665327013821d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:32:39.764539 10177 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b94cab2c92e410b93d665327013821d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b94cab2c92e410b93d665327013821d" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 41699 } }
I20260928 06:32:39.764927 10177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b94cab2c92e410b93d665327013821d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b94cab2c92e410b93d665327013821d. Latest consensus state: current_term: 1 leader_uuid: "6b94cab2c92e410b93d665327013821d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b94cab2c92e410b93d665327013821d" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 41699 } } }
I20260928 06:32:39.765038 10177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b94cab2c92e410b93d665327013821d [sys.catalog]: This master's current role is: LEADER
I20260928 06:32:39.764919 10178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b94cab2c92e410b93d665327013821d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b94cab2c92e410b93d665327013821d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b94cab2c92e410b93d665327013821d" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 41699 } } }
I20260928 06:32:39.765153 10178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b94cab2c92e410b93d665327013821d [sys.catalog]: This master's current role is: LEADER
Sep 28 06:32:39 dist-test-slave-d59m krb5kdc[10091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
W20260928 06:32:39.767418 10193 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6b94cab2c92e410b93d665327013821d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:32:39.767465 10193 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:32:39.767513 10181 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:32:39.767648 10181 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:32:39.769997 10181 catalog_manager.cc:1347] Generated new cluster ID: 0b0c045c88734f9cb2f8c85550306a0f
I20260928 06:32:39.770051 10181 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:32:39.772840 10175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:39.757685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36604 (local address 127.22.225.190:41699)
0928 06:32:39.757892 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:39.757902 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:39.757937 (+ 35us) server_negotiation.cc:407] Connection header received
0928 06:32:39.758479 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:39.758488 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:39.758550 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:39.758699 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:32:39.759530 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:39.761022 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:39.762124 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:39.762747 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:39.763159 (+ 412us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:39.765834 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:32:39.765838 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:32:39.765839 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:32:39.765852 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:32:39.769282 (+ 3430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:32:39.769684 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:32:39.769687 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:32:39.769687 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:32:39.769801 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:32:39.770003 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:32:39.770004 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:32:39.770004 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:32:39.771723 (+ 1719us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:32:39.771887 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:39.772462 (+ 575us) server_negotiation.cc:299] Negotiation successful
0928 06:32:39.772628 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":102,"threads_started":1}
I20260928 06:32:39.785499 10181 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:32:39.786326 10181 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:32:39.792095 10181 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b94cab2c92e410b93d665327013821d: Generated new TSK 0
I20260928 06:32:39.792317 10181 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.225.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab.
Entry for principal kudu/127.22.225.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab.
I20260928 06:32:39.825366 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.129:0
--local_ip_for_outbound_sockets=127.22.225.129
--webserver_interface=127.22.225.129
--webserver_port=0
--tserver_master_addrs=127.22.225.190:41699
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab
--principal=kudu/127.22.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:41853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:32:39.914575 10198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:32:39.914806 10198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:32:39.914846 10198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:32:39.915997 10198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:32:39.916065 10198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:32:39.916110 10198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.129
I20260928 06:32:39.917801 10198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:41853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.129.keytab
--principal=kudu/127.22.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.225.129
--webserver_port=0
--tserver_master_addrs=127.22.225.190:41699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.129
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:32:39.918084 10198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:32:39.918306 10198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:32:39.923722 10203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:39.923728 10206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:39.924409 10204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:32:39.924474 10198 server_base.cc:1034] running on GCE node
I20260928 06:32:39.924727 10198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:32:39.925005 10198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:32:39.926179 10198 hybrid_clock.cc:648] HybridClock initialized: now 1790577159926155 us; error 38 us; skew 500 ppm
Sep 28 06:32:39 dist-test-slave-d59m krb5kdc[10091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577159, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:32:39.929039 10198 init.cc:377] Logged in from keytab as kudu/127.22.225.129@KRBTEST.COM (short username kudu)
I20260928 06:32:39.929584 10198 webserver.cc:460] Webserver started at http://127.22.225.129:43871/ using document root <none> and password file <none>
I20260928 06:32:39.929819 10198 fs_manager.cc:359] Metadata directory not provided
I20260928 06:32:39.930095 10198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:32:39.930325 10198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:32:39.931387 10198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/minicluster-data/ts-0/data/instance:
uuid: "94c1f38c6bf54cdca97da7e22657adbc"
format_stamp: "Formatted at 2026-09-28 06:32:39 on dist-test-slave-d59m"
server_key: "29609643de67444ee993ba2fa4b3a3f9"
server_key_iv: "2433ccb89e12e87fa90e23cfd957caed"
server_key_version: "encryptionkey@0"
I20260928 06:32:39.931777 10198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/minicluster-data/ts-0/wal/instance:
uuid: "94c1f38c6bf54cdca97da7e22657adbc"
format_stamp: "Formatted at 2026-09-28 06:32:39 on dist-test-slave-d59m"
server_key: "29609643de67444ee993ba2fa4b3a3f9"
server_key_iv: "2433ccb89e12e87fa90e23cfd957caed"
server_key_version: "encryptionkey@0"
I20260928 06:32:39.933461 10198 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:32:39.934265 10213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:39.934423 10198 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:32:39.934509 10198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/minicluster-data/ts-0/wal
uuid: "94c1f38c6bf54cdca97da7e22657adbc"
format_stamp: "Formatted at 2026-09-28 06:32:39 on dist-test-slave-d59m"
server_key: "29609643de67444ee993ba2fa4b3a3f9"
server_key_iv: "2433ccb89e12e87fa90e23cfd957caed"
server_key_version: "encryptionkey@0"
I20260928 06:32:39.934577 10198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:32:39.957073 10198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:32:39.959398 10198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:32:39.959647 10198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:32:39.960031 10198 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:32:39.960508 10198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:32:39.960562 10198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:39.960618 10198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:32:39.960647 10198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:39.967635 10198 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.129:37659
I20260928 06:32:39.967684 10326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.129:37659 every 8 connection(s)
I20260928 06:32:39.968029 10198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
I20260928 06:32:39.970748 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 10198
I20260928 06:32:39.970888 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/minicluster-data/ts-0/wal/instance
I20260928 06:32:39.971050 23430 external_mini_cluster.cc:1428] Setting key 034abc69f44d6e64c3b990058e9989d3
Sep 28 06:32:39 dist-test-slave-d59m krb5kdc[10091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577159, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.129@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:32:39.979231 10175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:39.968831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:60787 (local address 127.22.225.190:41699)
0928 06:32:39.968977 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:39.968980 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:39.969158 (+ 178us) server_negotiation.cc:407] Connection header received
0928 06:32:39.969802 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:39.969803 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:39.969819 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:39.969899 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:32:39.971146 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:39.971837 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:39.973334 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:39.973683 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:39.973764 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:39.976303 (+ 2539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:32:39.976304 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:32:39.976305 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:32:39.976313 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:32:39.977795 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:32:39.978218 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:32:39.978219 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:32:39.978220 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:32:39.978280 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:32:39.978546 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:32:39.978547 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:32:39.978547 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:32:39.978665 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:32:39.978750 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:39.979048 (+ 298us) server_negotiation.cc:299] Negotiation successful
0928 06:32:39.979140 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260928 06:32:39.979426 10329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:39.968830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:41699 (local address 127.22.225.129:60787)
0928 06:32:39.969025 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:39.969076 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 06:32:39.969686 (+ 610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:32:39.969974 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:32:39.969979 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:32:39.970760 (+ 781us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:32:39.971039 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:39.971046 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:39.971942 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:39.971943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:39.973239 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:39.973242 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:39.973778 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:39.973779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:39.973927 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:39.974524 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:32:39.974534 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:32:39.976203 (+ 1669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:32:39.977903 (+ 1700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:32:39.977910 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:32:39.977913 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:32:39.978137 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:32:39.978357 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:32:39.978358 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:32:39.978358 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:32:39.978470 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:32:39.978749 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:32:39.978757 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:32:39.978905 (+ 148us) client_negotiation.cc:769] Sending connection context
0928 06:32:39.979012 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 06:32:39.979177 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":98,"threads_started":1}
I20260928 06:32:39.980131 10327 heartbeater.cc:347] Connected to a master server at 127.22.225.190:41699
WARNING: no policy specified for kudu/127.22.225.130@KRBTEST.COM; defaulting to no policy
I20260928 06:32:39.980257 10327 heartbeater.cc:464] Registering TS with master...
I20260928 06:32:39.980446 10327 heartbeater.cc:511] Master 127.22.225.190:41699 requested a full tablet report, sending...
I20260928 06:32:39.981014 10136 ts_manager.cc:194] Registered new tserver with Master: 94c1f38c6bf54cdca97da7e22657adbc (127.22.225.129:37659)
I20260928 06:32:39.981824 10136 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.225.129@KRBTEST.COM'} at 127.22.225.129:60787
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab.
Entry for principal kudu/127.22.225.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab.
I20260928 06:32:39.999814 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.130:0
--local_ip_for_outbound_sockets=127.22.225.130
--webserver_interface=127.22.225.130
--webserver_port=0
--tserver_master_addrs=127.22.225.190:41699
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab
--principal=kudu/127.22.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:41853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:32:40.085196 10332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:32:40.085433 10332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:32:40.085471 10332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:32:40.086668 10332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:32:40.086740 10332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:32:40.086782 10332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.130
I20260928 06:32:40.088250 10332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:41853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.130.keytab
--principal=kudu/127.22.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.225.130
--webserver_port=0
--tserver_master_addrs=127.22.225.190:41699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.130
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:32:40.088524 10332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:32:40.088786 10332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:32:40.093927 10337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:40.094054 10340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:40.093943 10338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:32:40.094147 10332 server_base.cc:1034] running on GCE node
I20260928 06:32:40.094367 10332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:32:40.094617 10332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:32:40.095820 10332 hybrid_clock.cc:648] HybridClock initialized: now 1790577160095798 us; error 41 us; skew 500 ppm
Sep 28 06:32:40 dist-test-slave-d59m krb5kdc[10091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577160, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:32:40.098754 10332 init.cc:377] Logged in from keytab as kudu/127.22.225.130@KRBTEST.COM (short username kudu)
I20260928 06:32:40.099274 10332 webserver.cc:460] Webserver started at http://127.22.225.130:34571/ using document root <none> and password file <none>
I20260928 06:32:40.099505 10332 fs_manager.cc:359] Metadata directory not provided
I20260928 06:32:40.099574 10332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:32:40.099736 10332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:32:40.100881 10332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/minicluster-data/ts-1/data/instance:
uuid: "4b811f70ecfb4568895b95d96ca3d249"
format_stamp: "Formatted at 2026-09-28 06:32:40 on dist-test-slave-d59m"
server_key: "03d787e2157834f8be3e8bc2cc7def54"
server_key_iv: "34d4b4a3109d3b7f5c040f6016517074"
server_key_version: "encryptionkey@0"
I20260928 06:32:40.101245 10332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/minicluster-data/ts-1/wal/instance:
uuid: "4b811f70ecfb4568895b95d96ca3d249"
format_stamp: "Formatted at 2026-09-28 06:32:40 on dist-test-slave-d59m"
server_key: "03d787e2157834f8be3e8bc2cc7def54"
server_key_iv: "34d4b4a3109d3b7f5c040f6016517074"
server_key_version: "encryptionkey@0"
I20260928 06:32:40.102665 10332 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:32:40.103477 10347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:40.103677 10332 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:32:40.103790 10332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/minicluster-data/ts-1/wal
uuid: "4b811f70ecfb4568895b95d96ca3d249"
format_stamp: "Formatted at 2026-09-28 06:32:40 on dist-test-slave-d59m"
server_key: "03d787e2157834f8be3e8bc2cc7def54"
server_key_iv: "34d4b4a3109d3b7f5c040f6016517074"
server_key_version: "encryptionkey@0"
I20260928 06:32:40.103866 10332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:32:40.120679 10332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:32:40.123160 10332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:32:40.123402 10332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:32:40.123654 10332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:32:40.123991 10332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:32:40.124047 10332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:40.124091 10332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:32:40.124119 10332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:40.130395 10332 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.130:34327
I20260928 06:32:40.130439 10460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.130:34327 every 8 connection(s)
I20260928 06:32:40.130774 10332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
I20260928 06:32:40.135085 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 10332
I20260928 06:32:40.135226 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/minicluster-data/ts-1/wal/instance
I20260928 06:32:40.135412 23430 external_mini_cluster.cc:1428] Setting key 29fdadc83f521ed29414a1e8e657c57e
Sep 28 06:32:40 dist-test-slave-d59m krb5kdc[10091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577160, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.130@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:32:40.142838 10175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:40.131629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:57045 (local address 127.22.225.190:41699)
0928 06:32:40.131689 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:40.131691 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:40.131931 (+ 240us) server_negotiation.cc:407] Connection header received
0928 06:32:40.132552 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:40.132553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:40.132564 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:40.132667 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:32:40.133940 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:40.134582 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:40.136317 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:40.136669 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:40.136760 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:40.139412 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:32:40.139414 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:32:40.139414 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:32:40.139423 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:32:40.141017 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:32:40.141457 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:32:40.141458 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:32:40.141458 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:32:40.141520 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:32:40.141831 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:32:40.141832 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:32:40.141832 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:32:40.141951 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:32:40.142052 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:40.142573 (+ 521us) server_negotiation.cc:299] Negotiation successful
0928 06:32:40.142723 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 06:32:40.142942 10463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:40.131587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:41699 (local address 127.22.225.130:57045)
0928 06:32:40.131787 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:40.131834 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 06:32:40.132450 (+ 616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:32:40.132747 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:32:40.132752 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:32:40.133507 (+ 755us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:32:40.133796 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:40.133803 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:40.134679 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:40.134679 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:40.136226 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:40.136229 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:40.136786 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:40.136787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:40.136938 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:40.137570 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:32:40.137582 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:32:40.139306 (+ 1724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:32:40.141109 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:32:40.141117 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:32:40.141120 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:32:40.141367 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:32:40.141630 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:32:40.141631 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:32:40.141631 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:32:40.141754 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:32:40.142126 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:32:40.142134 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:32:40.142356 (+ 222us) client_negotiation.cc:769] Sending connection context
0928 06:32:40.142518 (+ 162us) client_negotiation.cc:240] Negotiation successful
0928 06:32:40.142702 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":106,"threads_started":1}
I20260928 06:32:40.143765 10461 heartbeater.cc:347] Connected to a master server at 127.22.225.190:41699
I20260928 06:32:40.143877 10461 heartbeater.cc:464] Registering TS with master...
I20260928 06:32:40.144105 10461 heartbeater.cc:511] Master 127.22.225.190:41699 requested a full tablet report, sending...
I20260928 06:32:40.144572 10136 ts_manager.cc:194] Registered new tserver with Master: 4b811f70ecfb4568895b95d96ca3d249 (127.22.225.130:34327)
WARNING: no policy specified for kudu/127.22.225.131@KRBTEST.COM; defaulting to no policy
I20260928 06:32:40.145144 10136 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.225.130@KRBTEST.COM'} at 127.22.225.130:57045
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab.
Entry for principal kudu/127.22.225.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab.
I20260928 06:32:40.163627 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.131:0
--local_ip_for_outbound_sockets=127.22.225.131
--webserver_interface=127.22.225.131
--webserver_port=0
--tserver_master_addrs=127.22.225.190:41699
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab
--principal=kudu/127.22.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:41853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:32:40.254674 10466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:32:40.254920 10466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:32:40.254961 10466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:32:40.256141 10466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:32:40.256215 10466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:32:40.256263 10466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.131
I20260928 06:32:40.257969 10466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:41853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.131.keytab
--principal=kudu/127.22.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.225.131
--webserver_port=0
--tserver_master_addrs=127.22.225.190:41699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.131
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:32:40.258296 10466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:32:40.258538 10466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:32:40.263571 10472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:40.263572 10471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:40.263577 10474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:32:40.264027 10466 server_base.cc:1034] running on GCE node
I20260928 06:32:40.264271 10466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:32:40.264516 10466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:32:40.265753 10466 hybrid_clock.cc:648] HybridClock initialized: now 1790577160265667 us; error 54 us; skew 500 ppm
Sep 28 06:32:40 dist-test-slave-d59m krb5kdc[10091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577160, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:32:40.268523 10466 init.cc:377] Logged in from keytab as kudu/127.22.225.131@KRBTEST.COM (short username kudu)
I20260928 06:32:40.269052 10466 webserver.cc:460] Webserver started at http://127.22.225.131:40653/ using document root <none> and password file <none>
I20260928 06:32:40.269317 10466 fs_manager.cc:359] Metadata directory not provided
I20260928 06:32:40.269372 10466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:32:40.269500 10466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:32:40.270884 10466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/minicluster-data/ts-2/data/instance:
uuid: "14d171dd8de542eab5cbe6bc8eb1eaaf"
format_stamp: "Formatted at 2026-09-28 06:32:40 on dist-test-slave-d59m"
server_key: "2b798f74a2372e7c60510df80c24486c"
server_key_iv: "b13e560bca54ef238907d6dadf4cfa6e"
server_key_version: "encryptionkey@0"
I20260928 06:32:40.271287 10466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/minicluster-data/ts-2/wal/instance:
uuid: "14d171dd8de542eab5cbe6bc8eb1eaaf"
format_stamp: "Formatted at 2026-09-28 06:32:40 on dist-test-slave-d59m"
server_key: "2b798f74a2372e7c60510df80c24486c"
server_key_iv: "b13e560bca54ef238907d6dadf4cfa6e"
server_key_version: "encryptionkey@0"
I20260928 06:32:40.272917 10466 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.003s
I20260928 06:32:40.273842 10481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:40.274021 10466 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:32:40.274122 10466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/minicluster-data/ts-2/wal
uuid: "14d171dd8de542eab5cbe6bc8eb1eaaf"
format_stamp: "Formatted at 2026-09-28 06:32:40 on dist-test-slave-d59m"
server_key: "2b798f74a2372e7c60510df80c24486c"
server_key_iv: "b13e560bca54ef238907d6dadf4cfa6e"
server_key_version: "encryptionkey@0"
I20260928 06:32:40.274192 10466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:32:40.291560 10466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:32:40.294231 10466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:32:40.294404 10466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:32:40.294655 10466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:32:40.294984 10466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:32:40.295044 10466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:40.295086 10466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:32:40.295113 10466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:40.301419 10466 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.131:45727
I20260928 06:32:40.301465 10594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.131:45727 every 8 connection(s)
I20260928 06:32:40.301816 10466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
I20260928 06:32:40.309103 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 10466
I20260928 06:32:40.309207 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/minicluster-data/ts-2/wal/instance
I20260928 06:32:40.309391 23430 external_mini_cluster.cc:1428] Setting key 0153a55e881d04564a7b27d2260e6246
Sep 28 06:32:40 dist-test-slave-d59m krb5kdc[10091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577160, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.131@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:32:40.313273 10175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:40.302602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:53743 (local address 127.22.225.190:41699)
0928 06:32:40.302675 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:40.302678 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:40.302938 (+ 260us) server_negotiation.cc:407] Connection header received
0928 06:32:40.303590 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:40.303592 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:40.303601 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:40.303652 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:32:40.304934 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:40.305666 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:40.307214 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:40.307438 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:40.307500 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:40.310176 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:32:40.310178 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:32:40.310178 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:32:40.310187 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:32:40.311800 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:32:40.312272 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:32:40.312273 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:32:40.312273 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:32:40.312326 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:32:40.312623 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:32:40.312624 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:32:40.312625 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:32:40.312747 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:32:40.312802 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:40.313130 (+ 328us) server_negotiation.cc:299] Negotiation successful
0928 06:32:40.313191 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 06:32:40.313562 10597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:40.302578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:41699 (local address 127.22.225.131:53743)
0928 06:32:40.302790 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:40.302838 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 06:32:40.303470 (+ 632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:32:40.303734 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:32:40.303739 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:32:40.304491 (+ 752us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:32:40.304804 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:40.304810 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:40.305766 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:40.305766 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:40.307108 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:40.307111 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:40.307524 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:40.307525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:40.307667 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:40.308211 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:32:40.308221 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:32:40.310071 (+ 1850us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:32:40.311874 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:32:40.311881 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:32:40.311884 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:32:40.312198 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:32:40.312391 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:32:40.312391 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:32:40.312392 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:32:40.312524 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:32:40.312812 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:32:40.312818 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:32:40.312985 (+ 167us) client_negotiation.cc:769] Sending connection context
0928 06:32:40.313101 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 06:32:40.313269 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":118,"threads_started":1}
I20260928 06:32:40.314095 10595 heartbeater.cc:347] Connected to a master server at 127.22.225.190:41699
I20260928 06:32:40.314203 10595 heartbeater.cc:464] Registering TS with master...
I20260928 06:32:40.314392 10595 heartbeater.cc:511] Master 127.22.225.190:41699 requested a full tablet report, sending...
I20260928 06:32:40.314716 10136 ts_manager.cc:194] Registered new tserver with Master: 14d171dd8de542eab5cbe6bc8eb1eaaf (127.22.225.131:45727)
I20260928 06:32:40.315093 10136 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.225.131@KRBTEST.COM'} at 127.22.225.131:53743
I20260928 06:32:40.321807 23430 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:32:40.329460 10175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:40.323197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36614 (local address 127.22.225.190:41699)
0928 06:32:40.323268 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:40.323270 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:40.323373 (+ 103us) server_negotiation.cc:407] Connection header received
0928 06:32:40.323457 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:40.323459 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:40.323469 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:40.323525 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:32:40.324085 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:40.324623 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:40.325342 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:40.325552 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:40.325629 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:40.326824 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:32:40.326826 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:32:40.326827 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:32:40.326834 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:32:40.328181 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:32:40.328566 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:32:40.328567 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:32:40.328568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:32:40.328664 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:32:40.328873 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:32:40.328874 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:32:40.328875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:32:40.329036 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:32:40.329151 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:40.329278 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 06:32:40.329356 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 06:32:40.331233 10136 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36614:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:32:40.332894 10136 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:32:40.340104 10608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:40.335332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.131:45727 (local address 127.0.0.1:58880)
0928 06:32:40.335654 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:40.335667 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:32:40.335735 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:32:40.336257 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:32:40.336258 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:32:40.336261 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:32:40.336303 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:40.336305 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:40.337606 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:40.337607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:40.338837 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:40.338840 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:40.339779 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:40.339781 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:40.339866 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:40.339880 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 06:32:40.339961 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 06:32:40.340012 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":2,"thread_start_us":106,"threads_started":1}
I20260928 06:32:40.340124 10611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:40.335796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58880 (local address 127.22.225.131:45727)
0928 06:32:40.336084 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:40.336088 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:40.336098 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:32:40.336136 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:40.336138 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:40.336154 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:40.336376 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:32:40.336446 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:40.337501 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:40.338943 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:40.339670 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:40.339765 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:40.339812 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:40.339990 (+ 178us) server_negotiation.cc:299] Negotiation successful
0928 06:32:40.340033 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":90,"threads_started":1}
I20260928 06:32:40.340111 10607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:40.335311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.129:37659 (local address 127.0.0.1:53956)
0928 06:32:40.335513 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:40.335558 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 06:32:40.335622 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:32:40.335969 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:32:40.335972 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:32:40.335974 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:32:40.336044 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:40.336046 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:40.337458 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:40.337459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:40.338888 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:40.338890 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:40.339850 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:40.339851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:40.339919 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:40.339922 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:32:40.339995 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 06:32:40.340038 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":64,"threads_started":1}
I20260928 06:32:40.340312 10609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:40.335598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53956 (local address 127.22.225.129:37659)
0928 06:32:40.335792 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:40.335798 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:40.335809 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:32:40.335850 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:40.335852 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:40.335870 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:40.335946 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:32:40.336128 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:40.337354 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:40.338995 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:40.339680 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:40.339754 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:40.339813 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:40.340151 (+ 338us) server_negotiation.cc:299] Negotiation successful
0928 06:32:40.340197 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":87,"threads_started":1}
I20260928 06:32:40.340894 10606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:40.335517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.130:34327 (local address 127.0.0.1:33272)
0928 06:32:40.335965 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:40.335975 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:32:40.336087 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:32:40.336416 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:32:40.336416 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:32:40.336419 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:32:40.336461 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:40.336463 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:40.337895 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:40.337896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:40.339427 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:40.339430 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:40.340628 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:40.340629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:40.340700 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:40.340704 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:32:40.340794 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 06:32:40.340832 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":89,"threads_started":1}
I20260928 06:32:40.340938 10610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:40.335761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33272 (local address 127.22.225.130:34327)
0928 06:32:40.335936 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:40.335942 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:40.336076 (+ 134us) server_negotiation.cc:407] Connection header received
0928 06:32:40.336166 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:40.336169 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:40.336187 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:40.336263 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:32:40.336535 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:40.337742 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:40.339523 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:40.340453 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:40.340516 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:40.340594 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:40.340811 (+ 217us) server_negotiation.cc:299] Negotiation successful
0928 06:32:40.340853 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":92,"threads_started":1}
I20260928 06:32:40.341050 10261 tablet_service.cc:1467] Processing CreateTablet for tablet 4ac87f4288414660b2f57f1c94e5ddbf (DEFAULT_TABLE table=test-table [id=4b2c3291e3714613a851852d5d828c54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:32:40.341264 10529 tablet_service.cc:1467] Processing CreateTablet for tablet 4ac87f4288414660b2f57f1c94e5ddbf (DEFAULT_TABLE table=test-table [id=4b2c3291e3714613a851852d5d828c54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:32:40.341425 10261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ac87f4288414660b2f57f1c94e5ddbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:32:40.341545 10529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ac87f4288414660b2f57f1c94e5ddbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:32:40.344966 10612 tablet_bootstrap.cc:492] T 4ac87f4288414660b2f57f1c94e5ddbf P 94c1f38c6bf54cdca97da7e22657adbc: Bootstrap starting.
I20260928 06:32:40.345213 10395 tablet_service.cc:1467] Processing CreateTablet for tablet 4ac87f4288414660b2f57f1c94e5ddbf (DEFAULT_TABLE table=test-table [id=4b2c3291e3714613a851852d5d828c54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:32:40.345552 10395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ac87f4288414660b2f57f1c94e5ddbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:32:40.345757 10612 tablet_bootstrap.cc:654] T 4ac87f4288414660b2f57f1c94e5ddbf P 94c1f38c6bf54cdca97da7e22657adbc: Neither blocks nor log segments found. Creating new log.
I20260928 06:32:40.346220 10612 log.cc:824] T 4ac87f4288414660b2f57f1c94e5ddbf P 94c1f38c6bf54cdca97da7e22657adbc: Log is configured to *not* fsync() on all Append() calls
I20260928 06:32:40.347007 10612 tablet_bootstrap.cc:492] T 4ac87f4288414660b2f57f1c94e5ddbf P 94c1f38c6bf54cdca97da7e22657adbc: No bootstrap required, opened a new log
I20260928 06:32:40.347150 10612 ts_tablet_manager.cc:1397] T 4ac87f4288414660b2f57f1c94e5ddbf P 94c1f38c6bf54cdca97da7e22657adbc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:32:40.348503 10615 tablet_bootstrap.cc:492] T 4ac87f4288414660b2f57f1c94e5ddbf P 4b811f70ecfb4568895b95d96ca3d249: Bootstrap starting.
I20260928 06:32:40.348577 10613 tablet_bootstrap.cc:492] T 4ac87f4288414660b2f57f1c94e5ddbf P 14d171dd8de542eab5cbe6bc8eb1eaaf: Bootstrap starting.
I20260928 06:32:40.348682 10612 raft_consensus.cc:348] T 4ac87f4288414660b2f57f1c94e5ddbf P 94c1f38c6bf54cdca97da7e22657adbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94c1f38c6bf54cdca97da7e22657adbc" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 37659 } } peers { permanent_uuid: "4b811f70ecfb4568895b95d96ca3d249" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 34327 } } peers { permanent_uuid: "14d171dd8de542eab5cbe6bc8eb1eaaf" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 45727 } }
I20260928 06:32:40.348914 10612 raft_consensus.cc:374] T 4ac87f4288414660b2f57f1c94e5ddbf P 94c1f38c6bf54cdca97da7e22657adbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:32:40.348990 10612 raft_consensus.cc:729] T 4ac87f4288414660b2f57f1c94e5ddbf P 94c1f38c6bf54cdca97da7e22657adbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94c1f38c6bf54cdca97da7e22657adbc, State: Initialized, Role: FOLLOWER
I20260928 06:32:40.349143 10612 consensus_queue.cc:260] T 4ac87f4288414660b2f57f1c94e5ddbf P 94c1f38c6bf54cdca97da7e22657adbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94c1f38c6bf54cdca97da7e22657adbc" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 37659 } } peers { permanent_uuid: "4b811f70ecfb4568895b95d96ca3d249" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 34327 } } peers { permanent_uuid: "14d171dd8de542eab5cbe6bc8eb1eaaf" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 45727 } }
I20260928 06:32:40.349336 10615 tablet_bootstrap.cc:654] T 4ac87f4288414660b2f57f1c94e5ddbf P 4b811f70ecfb4568895b95d96ca3d249: Neither blocks nor log segments found. Creating new log.
I20260928 06:32:40.349346 10613 tablet_bootstrap.cc:654] T 4ac87f4288414660b2f57f1c94e5ddbf P 14d171dd8de542eab5cbe6bc8eb1eaaf: Neither blocks nor log segments found. Creating new log.
I20260928 06:32:40.349437 10612 ts_tablet_manager.cc:1428] T 4ac87f4288414660b2f57f1c94e5ddbf P 94c1f38c6bf54cdca97da7e22657adbc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:32:40.349596 10327 heartbeater.cc:503] Master 127.22.225.190:41699 was elected leader, sending a full tablet report...
I20260928 06:32:40.349838 10615 log.cc:824] T 4ac87f4288414660b2f57f1c94e5ddbf P 4b811f70ecfb4568895b95d96ca3d249: Log is configured to *not* fsync() on all Append() calls
I20260928 06:32:40.349924 10613 log.cc:824] T 4ac87f4288414660b2f57f1c94e5ddbf P 14d171dd8de542eab5cbe6bc8eb1eaaf: Log is configured to *not* fsync() on all Append() calls
I20260928 06:32:40.350570 10615 tablet_bootstrap.cc:492] T 4ac87f4288414660b2f57f1c94e5ddbf P 4b811f70ecfb4568895b95d96ca3d249: No bootstrap required, opened a new log
I20260928 06:32:40.350651 10615 ts_tablet_manager.cc:1397] T 4ac87f4288414660b2f57f1c94e5ddbf P 4b811f70ecfb4568895b95d96ca3d249: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:32:40.351114 10613 tablet_bootstrap.cc:492] T 4ac87f4288414660b2f57f1c94e5ddbf P 14d171dd8de542eab5cbe6bc8eb1eaaf: No bootstrap required, opened a new log
I20260928 06:32:40.351195 10613 ts_tablet_manager.cc:1397] T 4ac87f4288414660b2f57f1c94e5ddbf P 14d171dd8de542eab5cbe6bc8eb1eaaf: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260928 06:32:40.352223 10615 raft_consensus.cc:348] T 4ac87f4288414660b2f57f1c94e5ddbf P 4b811f70ecfb4568895b95d96ca3d249 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94c1f38c6bf54cdca97da7e22657adbc" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 37659 } } peers { permanent_uuid: "4b811f70ecfb4568895b95d96ca3d249" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 34327 } } peers { permanent_uuid: "14d171dd8de542eab5cbe6bc8eb1eaaf" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 45727 } }
I20260928 06:32:40.352385 10615 raft_consensus.cc:374] T 4ac87f4288414660b2f57f1c94e5ddbf P 4b811f70ecfb4568895b95d96ca3d249 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:32:40.352429 10615 raft_consensus.cc:729] T 4ac87f4288414660b2f57f1c94e5ddbf P 4b811f70ecfb4568895b95d96ca3d249 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b811f70ecfb4568895b95d96ca3d249, State: Initialized, Role: FOLLOWER
I20260928 06:32:40.352555 10615 consensus_queue.cc:260] T 4ac87f4288414660b2f57f1c94e5ddbf P 4b811f70ecfb4568895b95d96ca3d249 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94c1f38c6bf54cdca97da7e22657adbc" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 37659 } } peers { permanent_uuid: "4b811f70ecfb4568895b95d96ca3d249" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 34327 } } peers { permanent_uuid: "14d171dd8de542eab5cbe6bc8eb1eaaf" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 45727 } }
I20260928 06:32:40.352883 10615 ts_tablet_manager.cc:1428] T 4ac87f4288414660b2f57f1c94e5ddbf P 4b811f70ecfb4568895b95d96ca3d249: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:32:40.352910 10461 heartbeater.cc:503] Master 127.22.225.190:41699 was elected leader, sending a full tablet report...
I20260928 06:32:40.353103 10613 raft_consensus.cc:348] T 4ac87f4288414660b2f57f1c94e5ddbf P 14d171dd8de542eab5cbe6bc8eb1eaaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94c1f38c6bf54cdca97da7e22657adbc" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 37659 } } peers { permanent_uuid: "4b811f70ecfb4568895b95d96ca3d249" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 34327 } } peers { permanent_uuid: "14d171dd8de542eab5cbe6bc8eb1eaaf" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 45727 } }
I20260928 06:32:40.353277 10613 raft_consensus.cc:374] T 4ac87f4288414660b2f57f1c94e5ddbf P 14d171dd8de542eab5cbe6bc8eb1eaaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:32:40.353313 10613 raft_consensus.cc:729] T 4ac87f4288414660b2f57f1c94e5ddbf P 14d171dd8de542eab5cbe6bc8eb1eaaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14d171dd8de542eab5cbe6bc8eb1eaaf, State: Initialized, Role: FOLLOWER
I20260928 06:32:40.353413 10613 consensus_queue.cc:260] T 4ac87f4288414660b2f57f1c94e5ddbf P 14d171dd8de542eab5cbe6bc8eb1eaaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94c1f38c6bf54cdca97da7e22657adbc" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 37659 } } peers { permanent_uuid: "4b811f70ecfb4568895b95d96ca3d249" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 34327 } } peers { permanent_uuid: "14d171dd8de542eab5cbe6bc8eb1eaaf" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 45727 } }
I20260928 06:32:40.353672 10595 heartbeater.cc:503] Master 127.22.225.190:41699 was elected leader, sending a full tablet report...
I20260928 06:32:40.353884 10613 ts_tablet_manager.cc:1428] T 4ac87f4288414660b2f57f1c94e5ddbf P 14d171dd8de542eab5cbe6bc8eb1eaaf: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
W20260928 06:32:40.381378 10462 tablet.cc:2369] T 4ac87f4288414660b2f57f1c94e5ddbf P 4b811f70ecfb4568895b95d96ca3d249: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:32:40.468775 10328 tablet.cc:2369] T 4ac87f4288414660b2f57f1c94e5ddbf P 94c1f38c6bf54cdca97da7e22657adbc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:32:40.502393 10619 raft_consensus.cc:482] T 4ac87f4288414660b2f57f1c94e5ddbf P 4b811f70ecfb4568895b95d96ca3d249 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:32:40.502517 10619 raft_consensus.cc:504] T 4ac87f4288414660b2f57f1c94e5ddbf P 4b811f70ecfb4568895b95d96ca3d249 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94c1f38c6bf54cdca97da7e22657adbc" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 37659 } } peers { permanent_uuid: "4b811f70ecfb4568895b95d96ca3d249" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 34327 } } peers { permanent_uuid: "14d171dd8de542eab5cbe6bc8eb1eaaf" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 45727 } }
I20260928 06:32:40.502980 10619 leader_election.cc:288] T 4ac87f4288414660b2f57f1c94e5ddbf P 4b811f70ecfb4568895b95d96ca3d249 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94c1f38c6bf54cdca97da7e22657adbc (127.22.225.129:37659), 14d171dd8de542eab5cbe6bc8eb1eaaf (127.22.225.131:45727)
I20260928 06:32:40.506567 10611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:40.503313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:48837 (local address 127.22.225.131:45727)
0928 06:32:40.503358 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:40.503360 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:40.503373 (+ 13us) server_negotiation.cc:407] Connection header received
0928 06:32:40.503477 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:40.503478 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:40.503489 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:40.503546 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:32:40.503741 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:40.504519 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:40.505738 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:40.506203 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:40.506308 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:40.506318 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:40.506457 (+ 139us) server_negotiation.cc:299] Negotiation successful
0928 06:32:40.506498 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 06:32:40.506572 10463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:40.503194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.131:45727 (local address 127.22.225.130:48837)
0928 06:32:40.503327 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:40.503340 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:32:40.503391 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:32:40.503575 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:32:40.503576 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:32:40.503579 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:32:40.503652 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:40.503654 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:40.504679 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:40.504681 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:40.505636 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:40.505640 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:40.506286 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:40.506287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:40.506369 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:40.506373 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:32:40.506448 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 06:32:40.506488 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110}
I20260928 06:32:40.507053 10549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ac87f4288414660b2f57f1c94e5ddbf" candidate_uuid: "4b811f70ecfb4568895b95d96ca3d249" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14d171dd8de542eab5cbe6bc8eb1eaaf" is_pre_election: true
I20260928 06:32:40.507196 10609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:40.503419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:39447 (local address 127.22.225.129:37659)
0928 06:32:40.503478 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:40.503480 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:40.503593 (+ 113us) server_negotiation.cc:407] Connection header received
0928 06:32:40.503675 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:40.503676 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:40.503686 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:40.503737 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:32:40.503887 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:40.504570 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:40.506081 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:40.506608 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:40.506679 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:40.506688 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:40.507090 (+ 402us) server_negotiation.cc:299] Negotiation successful
0928 06:32:40.507132 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 06:32:40.507249 10549 raft_consensus.cc:2454] T 4ac87f4288414660b2f57f1c94e5ddbf P 14d171dd8de542eab5cbe6bc8eb1eaaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b811f70ecfb4568895b95d96ca3d249 in term 0.
I20260928 06:32:40.507323 10621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:40.503211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.129:37659 (local address 127.22.225.130:39447)
0928 06:32:40.503535 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:40.503546 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:32:40.503615 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:32:40.503760 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:32:40.503760 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:32:40.503762 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:32:40.503809 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:40.503811 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:40.504671 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:40.504672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:40.505983 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:40.505987 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:40.506936 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:40.506937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:40.506997 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:40.507000 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:32:40.507082 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 06:32:40.507122 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":74,"thread_start_us":107,"threads_started":1}
I20260928 06:32:40.507635 10349 leader_election.cc:302] T 4ac87f4288414660b2f57f1c94e5ddbf P 4b811f70ecfb4568895b95d96ca3d249 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14d171dd8de542eab5cbe6bc8eb1eaaf, 4b811f70ecfb4568895b95d96ca3d249; no voters:
I20260928 06:32:40.507756 10619 raft_consensus.cc:2781] T 4ac87f4288414660b2f57f1c94e5ddbf P 4b811f70ecfb4568895b95d96ca3d249 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:32:40.507819 10619 raft_consensus.cc:482] T 4ac87f4288414660b2f57f1c94e5ddbf P 4b811f70ecfb4568895b95d96ca3d249 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:32:40.507848 10619 raft_consensus.cc:3037] T 4ac87f4288414660b2f57f1c94e5ddbf P 4b811f70ecfb4568895b95d96ca3d249 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:32:40.507815 10281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ac87f4288414660b2f57f1c94e5ddbf" candidate_uuid: "4b811f70ecfb4568895b95d96ca3d249" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94c1f38c6bf54cdca97da7e22657adbc" is_pre_election: true
I20260928 06:32:40.507951 10281 raft_consensus.cc:2454] T 4ac87f4288414660b2f57f1c94e5ddbf P 94c1f38c6bf54cdca97da7e22657adbc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b811f70ecfb4568895b95d96ca3d249 in term 0.
I20260928 06:32:40.508672 10619 raft_consensus.cc:504] T 4ac87f4288414660b2f57f1c94e5ddbf P 4b811f70ecfb4568895b95d96ca3d249 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94c1f38c6bf54cdca97da7e22657adbc" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 37659 } } peers { permanent_uuid: "4b811f70ecfb4568895b95d96ca3d249" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 34327 } } peers { permanent_uuid: "14d171dd8de542eab5cbe6bc8eb1eaaf" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 45727 } }
I20260928 06:32:40.508823 10619 leader_election.cc:288] T 4ac87f4288414660b2f57f1c94e5ddbf P 4b811f70ecfb4568895b95d96ca3d249 [CANDIDATE]: Term 1 election: Requested vote from peers 94c1f38c6bf54cdca97da7e22657adbc (127.22.225.129:37659), 14d171dd8de542eab5cbe6bc8eb1eaaf (127.22.225.131:45727)
I20260928 06:32:40.508966 10281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ac87f4288414660b2f57f1c94e5ddbf" candidate_uuid: "4b811f70ecfb4568895b95d96ca3d249" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94c1f38c6bf54cdca97da7e22657adbc"
I20260928 06:32:40.509002 10549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ac87f4288414660b2f57f1c94e5ddbf" candidate_uuid: "4b811f70ecfb4568895b95d96ca3d249" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14d171dd8de542eab5cbe6bc8eb1eaaf"
I20260928 06:32:40.509054 10281 raft_consensus.cc:3037] T 4ac87f4288414660b2f57f1c94e5ddbf P 94c1f38c6bf54cdca97da7e22657adbc [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:32:40.509114 10549 raft_consensus.cc:3037] T 4ac87f4288414660b2f57f1c94e5ddbf P 14d171dd8de542eab5cbe6bc8eb1eaaf [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:32:40.509891 10549 raft_consensus.cc:2454] T 4ac87f4288414660b2f57f1c94e5ddbf P 14d171dd8de542eab5cbe6bc8eb1eaaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b811f70ecfb4568895b95d96ca3d249 in term 1.
I20260928 06:32:40.509974 10281 raft_consensus.cc:2454] T 4ac87f4288414660b2f57f1c94e5ddbf P 94c1f38c6bf54cdca97da7e22657adbc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b811f70ecfb4568895b95d96ca3d249 in term 1.
I20260928 06:32:40.510062 10349 leader_election.cc:302] T 4ac87f4288414660b2f57f1c94e5ddbf P 4b811f70ecfb4568895b95d96ca3d249 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14d171dd8de542eab5cbe6bc8eb1eaaf, 4b811f70ecfb4568895b95d96ca3d249; no voters:
I20260928 06:32:40.510192 10619 raft_consensus.cc:2781] T 4ac87f4288414660b2f57f1c94e5ddbf P 4b811f70ecfb4568895b95d96ca3d249 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:32:40.510722 10619 raft_consensus.cc:686] T 4ac87f4288414660b2f57f1c94e5ddbf P 4b811f70ecfb4568895b95d96ca3d249 [term 1 LEADER]: Becoming Leader. State: Replica: 4b811f70ecfb4568895b95d96ca3d249, State: Running, Role: LEADER
I20260928 06:32:40.510850 10619 consensus_queue.cc:237] T 4ac87f4288414660b2f57f1c94e5ddbf P 4b811f70ecfb4568895b95d96ca3d249 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94c1f38c6bf54cdca97da7e22657adbc" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 37659 } } peers { permanent_uuid: "4b811f70ecfb4568895b95d96ca3d249" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 34327 } } peers { permanent_uuid: "14d171dd8de542eab5cbe6bc8eb1eaaf" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 45727 } }
I20260928 06:32:40.511807 10136 catalog_manager.cc:5495] T 4ac87f4288414660b2f57f1c94e5ddbf P 4b811f70ecfb4568895b95d96ca3d249 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b811f70ecfb4568895b95d96ca3d249 (127.22.225.130). New cstate: current_term: 1 leader_uuid: "4b811f70ecfb4568895b95d96ca3d249" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94c1f38c6bf54cdca97da7e22657adbc" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 37659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b811f70ecfb4568895b95d96ca3d249" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 34327 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14d171dd8de542eab5cbe6bc8eb1eaaf" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 45727 } health_report { overall_health: UNKNOWN } } }
I20260928 06:32:40.525480 10610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:40.522640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33280 (local address 127.22.225.130:34327)
0928 06:32:40.522704 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:40.522706 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:40.522714 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:32:40.522757 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:40.522758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:40.522780 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:40.522866 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:32:40.522949 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:40.523823 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:40.524594 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:40.524907 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:40.524973 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:40.525120 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:32:40.525254 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:32:40.525338 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:40.525354 (+ 16us) server_negotiation.cc:299] Negotiation successful
0928 06:32:40.525376 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 06:32:40.527498 10549 raft_consensus.cc:1261] T 4ac87f4288414660b2f57f1c94e5ddbf P 14d171dd8de542eab5cbe6bc8eb1eaaf [term 1 FOLLOWER]: Refusing update from remote peer 4b811f70ecfb4568895b95d96ca3d249: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:32:40.527643 10281 raft_consensus.cc:1261] T 4ac87f4288414660b2f57f1c94e5ddbf P 94c1f38c6bf54cdca97da7e22657adbc [term 1 FOLLOWER]: Refusing update from remote peer 4b811f70ecfb4568895b95d96ca3d249: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:32:40.527865 10619 consensus_queue.cc:1035] T 4ac87f4288414660b2f57f1c94e5ddbf P 4b811f70ecfb4568895b95d96ca3d249 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d171dd8de542eab5cbe6bc8eb1eaaf" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 45727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:32:40.528036 10622 consensus_queue.cc:1035] T 4ac87f4288414660b2f57f1c94e5ddbf P 4b811f70ecfb4568895b95d96ca3d249 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94c1f38c6bf54cdca97da7e22657adbc" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 37659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:32:40.530869 10625 mvcc.cc:205] Tried to move back new op lower bound from 7334204049517707264 to 7334204049453056000. Current Snapshot: MvccSnapshot[applied={T|T < 7334204049517707264}]
I20260928 06:32:40.531240 10628 mvcc.cc:205] Tried to move back new op lower bound from 7334204049517707264 to 7334204049453056000. Current Snapshot: MvccSnapshot[applied={T|T < 7334204049517707264}]
I20260928 06:32:40.533064 10136 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:32:40.533174 10136 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:32:40.533905 10136 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6b94cab2c92e410b93d665327013821d: Sending DeleteTablet for 3 replicas of tablet 4ac87f4288414660b2f57f1c94e5ddbf
I20260928 06:32:40.534278 10261 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ac87f4288414660b2f57f1c94e5ddbf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:32:40 UTC) from {username='kudu', principal='kudu/127.22.225.190@KRBTEST.COM'} at 127.0.0.1:53956
I20260928 06:32:40.534287 10529 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ac87f4288414660b2f57f1c94e5ddbf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:32:40 UTC) from {username='kudu', principal='kudu/127.22.225.190@KRBTEST.COM'} at 127.0.0.1:58880
I20260928 06:32:40.534559 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 10198
I20260928 06:32:40.534546 10395 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ac87f4288414660b2f57f1c94e5ddbf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:32:40 UTC) from {username='kudu', principal='kudu/127.22.225.190@KRBTEST.COM'} at 127.0.0.1:33272
I20260928 06:32:40.535279 10636 tablet_replica.cc:330] T 4ac87f4288414660b2f57f1c94e5ddbf P 4b811f70ecfb4568895b95d96ca3d249: stopping tablet replica
I20260928 06:32:40.535398 10636 raft_consensus.cc:2229] T 4ac87f4288414660b2f57f1c94e5ddbf P 4b811f70ecfb4568895b95d96ca3d249 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:32:40.535507 10636 raft_consensus.cc:2258] T 4ac87f4288414660b2f57f1c94e5ddbf P 4b811f70ecfb4568895b95d96ca3d249 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:32:40.535964 10636 ts_tablet_manager.cc:1905] T 4ac87f4288414660b2f57f1c94e5ddbf P 4b811f70ecfb4568895b95d96ca3d249: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:32:40.538281 10636 ts_tablet_manager.cc:1918] T 4ac87f4288414660b2f57f1c94e5ddbf P 4b811f70ecfb4568895b95d96ca3d249: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:32:40.538421 10636 log.cc:1196] T 4ac87f4288414660b2f57f1c94e5ddbf P 4b811f70ecfb4568895b95d96ca3d249: Deleting WAL directory at /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/minicluster-data/ts-1/wal/wals/4ac87f4288414660b2f57f1c94e5ddbf
I20260928 06:32:40.538775 10636 ts_tablet_manager.cc:1939] T 4ac87f4288414660b2f57f1c94e5ddbf P 4b811f70ecfb4568895b95d96ca3d249: Deleting consensus metadata
I20260928 06:32:40.538877 10634 tablet_replica.cc:330] T 4ac87f4288414660b2f57f1c94e5ddbf P 14d171dd8de542eab5cbe6bc8eb1eaaf: stopping tablet replica
I20260928 06:32:40.539001 10634 raft_consensus.cc:2229] T 4ac87f4288414660b2f57f1c94e5ddbf P 14d171dd8de542eab5cbe6bc8eb1eaaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:32:40.539135 10634 raft_consensus.cc:2258] T 4ac87f4288414660b2f57f1c94e5ddbf P 14d171dd8de542eab5cbe6bc8eb1eaaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:32:40.539525 10122 catalog_manager.cc:4841] TS 4b811f70ecfb4568895b95d96ca3d249 (127.22.225.130:34327): tablet 4ac87f4288414660b2f57f1c94e5ddbf (table test-table [id=4b2c3291e3714613a851852d5d828c54]) successfully deleted
I20260928 06:32:40.539726 10634 ts_tablet_manager.cc:1905] T 4ac87f4288414660b2f57f1c94e5ddbf P 14d171dd8de542eab5cbe6bc8eb1eaaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:32:40.541177 10634 ts_tablet_manager.cc:1918] T 4ac87f4288414660b2f57f1c94e5ddbf P 14d171dd8de542eab5cbe6bc8eb1eaaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:32:40.541266 10634 log.cc:1196] T 4ac87f4288414660b2f57f1c94e5ddbf P 14d171dd8de542eab5cbe6bc8eb1eaaf: Deleting WAL directory at /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576899143097-23430-0/minicluster-data/ts-2/wal/wals/4ac87f4288414660b2f57f1c94e5ddbf
I20260928 06:32:40.541518 10634 ts_tablet_manager.cc:1939] T 4ac87f4288414660b2f57f1c94e5ddbf P 14d171dd8de542eab5cbe6bc8eb1eaaf: Deleting consensus metadata
W20260928 06:32:40.544763 10122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.225.129:37659 (error 108)
I20260928 06:32:40.544955 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 10332
I20260928 06:32:40.546208 10606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:40.546057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.129:37659 (local address 127.0.0.1:53972)
0928 06:32:40.546116 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:40.546157 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.129:37659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 06:32:40.546329 10122 catalog_manager.cc:4584] TS 94c1f38c6bf54cdca97da7e22657adbc (127.22.225.129:37659): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4ac87f4288414660b2f57f1c94e5ddbf: Network error: Client connection negotiation failed: client connection to 127.22.225.129:37659: connect: Connection refused (error 111)
I20260928 06:32:40.548794 10121 catalog_manager.cc:4841] TS 14d171dd8de542eab5cbe6bc8eb1eaaf (127.22.225.131:45727): tablet 4ac87f4288414660b2f57f1c94e5ddbf (table test-table [id=4b2c3291e3714613a851852d5d828c54]) successfully deleted
I20260928 06:32:40.551795 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 10466
I20260928 06:32:40.556035 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 10104
2026-09-28T06:32:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2166 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T06:32:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:32:40Z Disabled control of system clock
I20260928 06:32:40.585036 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576899143097-23430-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576899143097-23430-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576899143097-23430-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:37703
--webserver_interface=127.22.225.190
--webserver_port=0
--builtin_ntp_servers=127.22.225.148:33587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.225.190:37703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 06:32:40.667857 10641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:32:40.668135 10641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:32:40.668176 10641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:32:40.669528 10641 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:32:40.669596 10641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:32:40.669620 10641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:32:40.669652 10641 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:32:40.669677 10641 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:32:40.671084 10641 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:33587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576899143097-23430-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576899143097-23430-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.225.190:37703
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:37703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576899143097-23430-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:32:40.671391 10641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:32:40.671695 10641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:32:40.676585 10646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:40.676563 10649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:32:40.676848 10641 server_base.cc:1034] running on GCE node
W20260928 06:32:40.676558 10647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:32:40.677124 10641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:32:40.677455 10641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:32:40.678644 10641 hybrid_clock.cc:648] HybridClock initialized: now 1790577160678633 us; error 60 us; skew 500 ppm
I20260928 06:32:40.680116 10641 webserver.cc:460] Webserver started at http://127.22.225.190:46263/ using document root <none> and password file <none>
I20260928 06:32:40.680414 10641 fs_manager.cc:359] Metadata directory not provided
I20260928 06:32:40.680487 10641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:32:40.680683 10641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:32:40.681798 10641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576899143097-23430-0/minicluster-data/master-0/data/instance:
uuid: "c96fa69ee475490689ac820066d8c9ca"
format_stamp: "Formatted at 2026-09-28 06:32:40 on dist-test-slave-d59m"
server_key: "814a4979558677017e2ea3174bcddf0e"
server_key_iv: "4634d6d74213bb7579337d527cd9ff19"
server_key_version: "encryptionkey@0"
I20260928 06:32:40.682175 10641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576899143097-23430-0/minicluster-data/master-0/wal/instance:
uuid: "c96fa69ee475490689ac820066d8c9ca"
format_stamp: "Formatted at 2026-09-28 06:32:40 on dist-test-slave-d59m"
server_key: "814a4979558677017e2ea3174bcddf0e"
server_key_iv: "4634d6d74213bb7579337d527cd9ff19"
server_key_version: "encryptionkey@0"
I20260928 06:32:40.683553 10641 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:32:40.684564 10655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:40.684732 10641 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:32:40.684863 10641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576899143097-23430-0/minicluster-data/master-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576899143097-23430-0/minicluster-data/master-0/wal
uuid: "c96fa69ee475490689ac820066d8c9ca"
format_stamp: "Formatted at 2026-09-28 06:32:40 on dist-test-slave-d59m"
server_key: "814a4979558677017e2ea3174bcddf0e"
server_key_iv: "4634d6d74213bb7579337d527cd9ff19"
server_key_version: "encryptionkey@0"
I20260928 06:32:40.684948 10641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576899143097-23430-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576899143097-23430-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576899143097-23430-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:32:40.699935 10641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:32:40.700295 10641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:32:40.700456 10641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:32:40.705502 10641 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.190:37703
I20260928 06:32:40.705531 10707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.190:37703 every 8 connection(s)
I20260928 06:32:40.705948 10641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
I20260928 06:32:40.706996 10708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:32:40.709558 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 10641
I20260928 06:32:40.709671 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576899143097-23430-0/minicluster-data/master-0/wal/instance
I20260928 06:32:40.709710 10708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c96fa69ee475490689ac820066d8c9ca: Bootstrap starting.
I20260928 06:32:40.709896 23430 external_mini_cluster.cc:1428] Setting key ab6063537fac5d2b5404893d61e7f524
I20260928 06:32:40.710449 10708 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c96fa69ee475490689ac820066d8c9ca: Neither blocks nor log segments found. Creating new log.
I20260928 06:32:40.710827 10708 log.cc:824] T 00000000000000000000000000000000 P c96fa69ee475490689ac820066d8c9ca: Log is configured to *not* fsync() on all Append() calls
I20260928 06:32:40.711571 10708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c96fa69ee475490689ac820066d8c9ca: No bootstrap required, opened a new log
I20260928 06:32:40.713951 10708 raft_consensus.cc:348] T 00000000000000000000000000000000 P c96fa69ee475490689ac820066d8c9ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c96fa69ee475490689ac820066d8c9ca" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 37703 } }
I20260928 06:32:40.714128 10708 raft_consensus.cc:374] T 00000000000000000000000000000000 P c96fa69ee475490689ac820066d8c9ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:32:40.714162 10708 raft_consensus.cc:729] T 00000000000000000000000000000000 P c96fa69ee475490689ac820066d8c9ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c96fa69ee475490689ac820066d8c9ca, State: Initialized, Role: FOLLOWER
I20260928 06:32:40.714319 10708 consensus_queue.cc:260] T 00000000000000000000000000000000 P c96fa69ee475490689ac820066d8c9ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c96fa69ee475490689ac820066d8c9ca" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 37703 } }
I20260928 06:32:40.714387 10708 raft_consensus.cc:388] T 00000000000000000000000000000000 P c96fa69ee475490689ac820066d8c9ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:32:40.714414 10708 raft_consensus.cc:482] T 00000000000000000000000000000000 P c96fa69ee475490689ac820066d8c9ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:32:40.714452 10708 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c96fa69ee475490689ac820066d8c9ca [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:32:40.715095 10708 raft_consensus.cc:504] T 00000000000000000000000000000000 P c96fa69ee475490689ac820066d8c9ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c96fa69ee475490689ac820066d8c9ca" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 37703 } }
I20260928 06:32:40.715227 10708 leader_election.cc:302] T 00000000000000000000000000000000 P c96fa69ee475490689ac820066d8c9ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c96fa69ee475490689ac820066d8c9ca; no voters:
I20260928 06:32:40.715432 10708 leader_election.cc:288] T 00000000000000000000000000000000 P c96fa69ee475490689ac820066d8c9ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:32:40.715724 10708 sys_catalog.cc:564] T 00000000000000000000000000000000 P c96fa69ee475490689ac820066d8c9ca [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:32:40.717191 10711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:40.710476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45398 (local address 127.22.225.190:37703)
0928 06:32:40.710843 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:40.710853 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:40.710892 (+ 39us) server_negotiation.cc:407] Connection header received
0928 06:32:40.711491 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:40.711502 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:40.711583 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:40.711782 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:32:40.712054 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:40.713541 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:40.714703 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:40.715354 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:40.715484 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:40.716248 (+ 764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:32:40.716252 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:32:40.716253 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:32:40.716461 (+ 208us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:32:40.716504 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:32:40.716518 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:32:40.716605 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:40.716821 (+ 216us) server_negotiation.cc:299] Negotiation successful
0928 06:32:40.716922 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":97,"threads_started":1}
I20260928 06:32:40.717258 10714 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c96fa69ee475490689ac820066d8c9ca [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:32:40.717551 10714 raft_consensus.cc:686] T 00000000000000000000000000000000 P c96fa69ee475490689ac820066d8c9ca [term 1 LEADER]: Becoming Leader. State: Replica: c96fa69ee475490689ac820066d8c9ca, State: Running, Role: LEADER
I20260928 06:32:40.717638 10714 consensus_queue.cc:237] T 00000000000000000000000000000000 P c96fa69ee475490689ac820066d8c9ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c96fa69ee475490689ac820066d8c9ca" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 37703 } }
I20260928 06:32:40.718185 10713 sys_catalog.cc:455] T 00000000000000000000000000000000 P c96fa69ee475490689ac820066d8c9ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c96fa69ee475490689ac820066d8c9ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c96fa69ee475490689ac820066d8c9ca" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 37703 } } }
I20260928 06:32:40.718267 10713 sys_catalog.cc:458] T 00000000000000000000000000000000 P c96fa69ee475490689ac820066d8c9ca [sys.catalog]: This master's current role is: LEADER
I20260928 06:32:40.718228 10726 sys_catalog.cc:455] T 00000000000000000000000000000000 P c96fa69ee475490689ac820066d8c9ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader c96fa69ee475490689ac820066d8c9ca. Latest consensus state: current_term: 1 leader_uuid: "c96fa69ee475490689ac820066d8c9ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c96fa69ee475490689ac820066d8c9ca" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 37703 } } }
I20260928 06:32:40.718830 10726 sys_catalog.cc:458] T 00000000000000000000000000000000 P c96fa69ee475490689ac820066d8c9ca [sys.catalog]: This master's current role is: LEADER
W20260928 06:32:40.718895 10725 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c96fa69ee475490689ac820066d8c9ca: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:32:40.718947 10725 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:32:40.719089 10730 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:32:40.719271 10730 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:32:40.720789 10730 catalog_manager.cc:1347] Generated new cluster ID: 5615a604fdc24951945ef94c0c43921e
I20260928 06:32:40.720840 10730 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:32:40.729581 10730 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:32:40.730345 10730 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:32:40.735363 10730 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c96fa69ee475490689ac820066d8c9ca: Generated new TSK 0
I20260928 06:32:40.735541 10730 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:32:40.743050 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576899143097-23430-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576899143097-23430-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576899143097-23430-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.129:0
--local_ip_for_outbound_sockets=127.22.225.129
--webserver_interface=127.22.225.129
--webserver_port=0
--tserver_master_addrs=127.22.225.190:37703
--builtin_ntp_servers=127.22.225.148:33587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 06:32:40.859510 10732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:32:40.859800 10732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:32:40.859845 10732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:32:40.861527 10732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:32:40.861575 10732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:32:40.861609 10732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.129
I20260928 06:32:40.863559 10732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:33587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576899143097-23430-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576899143097-23430-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.225.129
--webserver_port=0
--tserver_master_addrs=127.22.225.190:37703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.129
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576899143097-23430-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:32:40.863826 10732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:32:40.864163 10732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:32:40.869696 10738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:40.869737 10737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:32:40.869911 10732 server_base.cc:1034] running on GCE node
W20260928 06:32:40.869946 10740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:32:40.870220 10732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:32:40.870504 10732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:32:40.871671 10732 hybrid_clock.cc:648] HybridClock initialized: now 1790577160871633 us; error 52 us; skew 500 ppm
I20260928 06:32:40.873132 10732 webserver.cc:460] Webserver started at http://127.22.225.129:36589/ using document root <none> and password file <none>
I20260928 06:32:40.873397 10732 fs_manager.cc:359] Metadata directory not provided
I20260928 06:32:40.873462 10732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:32:40.873592 10732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:32:40.874963 10732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576899143097-23430-0/minicluster-data/ts-0/data/instance:
uuid: "5735acfb6e2a48deadf6b4a52bfe214c"
format_stamp: "Formatted at 2026-09-28 06:32:40 on dist-test-slave-d59m"
server_key: "ef478a2b6b8528c96fc0ef5778aaea71"
server_key_iv: "1fda82be6c13712d25b105e16bfb87a0"
server_key_version: "encryptionkey@0"
I20260928 06:32:40.875365 10732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576899143097-23430-0/minicluster-data/ts-0/wal/instance:
uuid: "5735acfb6e2a48deadf6b4a52bfe214c"
format_stamp: "Formatted at 2026-09-28 06:32:40 on dist-test-slave-d59m"
server_key: "ef478a2b6b8528c96fc0ef5778aaea71"
server_key_iv: "1fda82be6c13712d25b105e16bfb87a0"
server_key_version: "encryptionkey@0"
I20260928 06:32:40.876986 10732 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260928 06:32:40.877861 10746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:40.878034 10732 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:32:40.878145 10732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576899143097-23430-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576899143097-23430-0/minicluster-data/ts-0/wal
uuid: "5735acfb6e2a48deadf6b4a52bfe214c"
format_stamp: "Formatted at 2026-09-28 06:32:40 on dist-test-slave-d59m"
server_key: "ef478a2b6b8528c96fc0ef5778aaea71"
server_key_iv: "1fda82be6c13712d25b105e16bfb87a0"
server_key_version: "encryptionkey@0"
I20260928 06:32:40.878232 10732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576899143097-23430-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576899143097-23430-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576899143097-23430-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:32:40.884739 10732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:32:40.885032 10732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:32:40.885181 10732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:32:40.885404 10732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:32:40.885718 10732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:32:40.885773 10732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:40.885814 10732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:32:40.885842 10732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:40.892230 10732 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.129:36607
I20260928 06:32:40.892261 10859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.129:36607 every 8 connection(s)
I20260928 06:32:40.892709 10732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576899143097-23430-0/minicluster-data/ts-0/data/info.pb
I20260928 06:32:40.898564 10711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:40.893474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.129:43607 (local address 127.22.225.190:37703)
0928 06:32:40.893519 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:40.893521 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:40.893832 (+ 311us) server_negotiation.cc:407] Connection header received
0928 06:32:40.894458 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:40.894460 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:40.894473 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:40.894525 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:32:40.895401 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:40.895931 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:40.897026 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:40.897309 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:40.897389 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:40.898319 (+ 930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:32:40.898320 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:32:40.898320 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:32:40.898324 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:32:40.898339 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:32:40.898343 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:32:40.898389 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:40.898483 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 06:32:40.898505 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:32:40.898810 10862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:40.893458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:37703 (local address 127.22.225.129:43607)
0928 06:32:40.893663 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:40.893718 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 06:32:40.894337 (+ 619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:32:40.894596 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:32:40.894601 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:32:40.894611 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:32:40.895284 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:40.895290 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:40.896041 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:40.896042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:40.896934 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:40.896937 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:40.897431 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:40.897433 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:40.897598 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:40.898162 (+ 564us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:32:40.898163 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:32:40.898202 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:32:40.898229 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:32:40.898404 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:32:40.898409 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:32:40.898413 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:32:40.898478 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 06:32:40.898583 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":93,"threads_started":1}
I20260928 06:32:40.899287 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 10732
I20260928 06:32:40.899307 10860 heartbeater.cc:347] Connected to a master server at 127.22.225.190:37703
I20260928 06:32:40.899384 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576899143097-23430-0/minicluster-data/ts-0/wal/instance
I20260928 06:32:40.899387 10860 heartbeater.cc:464] Registering TS with master...
I20260928 06:32:40.899567 10860 heartbeater.cc:511] Master 127.22.225.190:37703 requested a full tablet report, sending...
I20260928 06:32:40.899592 23430 external_mini_cluster.cc:1428] Setting key c56da00141af02e345eac57d5280c05b
I20260928 06:32:40.900089 10672 ts_manager.cc:194] Registered new tserver with Master: 5735acfb6e2a48deadf6b4a52bfe214c (127.22.225.129:36607)
I20260928 06:32:40.900863 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576899143097-23430-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576899143097-23430-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576899143097-23430-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.130:0
--local_ip_for_outbound_sockets=127.22.225.130
--webserver_interface=127.22.225.130
--webserver_port=0
--tserver_master_addrs=127.22.225.190:37703
--builtin_ntp_servers=127.22.225.148:33587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 06:32:40.901060 10672 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.225.129:43607
W20260928 06:32:40.992421 10863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:32:40.992702 10863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:32:40.992743 10863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:32:40.993919 10863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:32:40.993984 10863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:32:40.994027 10863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.130
I20260928 06:32:40.995389 10863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:33587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576899143097-23430-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576899143097-23430-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.225.130
--webserver_port=0
--tserver_master_addrs=127.22.225.190:37703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.130
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576899143097-23430-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:32:40.995662 10863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:32:40.995976 10863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:32:41.001055 10868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:41.001240 10869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:41.001042 10871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:32:41.001401 10863 server_base.cc:1034] running on GCE node
I20260928 06:32:41.001626 10863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:32:41.001920 10863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:32:41.003093 10863 hybrid_clock.cc:648] HybridClock initialized: now 1790577161003066 us; error 42 us; skew 500 ppm
I20260928 06:32:41.004382 10863 webserver.cc:460] Webserver started at http://127.22.225.130:41659/ using document root <none> and password file <none>
I20260928 06:32:41.004732 10863 fs_manager.cc:359] Metadata directory not provided
I20260928 06:32:41.004828 10863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:32:41.005020 10863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:32:41.006134 10863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576899143097-23430-0/minicluster-data/ts-1/data/instance:
uuid: "84f00bd875e448bfb4dd02794539d1b2"
format_stamp: "Formatted at 2026-09-28 06:32:41 on dist-test-slave-d59m"
server_key: "8d6655fc8e077cbd7c5fc5156363d8df"
server_key_iv: "56a6e24cf1ac1a4ccfcb480204f5b2f9"
server_key_version: "encryptionkey@0"
I20260928 06:32:41.006523 10863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576899143097-23430-0/minicluster-data/ts-1/wal/instance:
uuid: "84f00bd875e448bfb4dd02794539d1b2"
format_stamp: "Formatted at 2026-09-28 06:32:41 on dist-test-slave-d59m"
server_key: "8d6655fc8e077cbd7c5fc5156363d8df"
server_key_iv: "56a6e24cf1ac1a4ccfcb480204f5b2f9"
server_key_version: "encryptionkey@0"
I20260928 06:32:41.008071 10863 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:32:41.008890 10877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:41.009089 10863 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:32:41.009205 10863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576899143097-23430-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576899143097-23430-0/minicluster-data/ts-1/wal
uuid: "84f00bd875e448bfb4dd02794539d1b2"
format_stamp: "Formatted at 2026-09-28 06:32:41 on dist-test-slave-d59m"
server_key: "8d6655fc8e077cbd7c5fc5156363d8df"
server_key_iv: "56a6e24cf1ac1a4ccfcb480204f5b2f9"
server_key_version: "encryptionkey@0"
I20260928 06:32:41.009301 10863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576899143097-23430-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576899143097-23430-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576899143097-23430-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:32:41.048812 10863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:32:41.049158 10863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:32:41.049317 10863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:32:41.049593 10863 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:32:41.049959 10863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:32:41.050019 10863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:41.050068 10863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:32:41.050097 10863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:41.057029 10863 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.130:35279
I20260928 06:32:41.057061 10990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.130:35279 every 8 connection(s)
I20260928 06:32:41.057436 10863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576899143097-23430-0/minicluster-data/ts-1/data/info.pb
I20260928 06:32:41.063529 10711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:41.058223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.130:33379 (local address 127.22.225.190:37703)
0928 06:32:41.058302 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:41.058304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:41.058582 (+ 278us) server_negotiation.cc:407] Connection header received
0928 06:32:41.059221 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:41.059222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:41.059233 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:41.059276 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:32:41.060109 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:41.060660 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:41.062086 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:41.062279 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:41.062343 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:41.063274 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:32:41.063275 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:32:41.063276 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:32:41.063280 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:32:41.063293 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:32:41.063297 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:32:41.063343 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:41.063447 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 06:32:41.063469 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 06:32:41.063761 10993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:41.058208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:37703 (local address 127.22.225.130:33379)
0928 06:32:41.058436 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:41.058487 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 06:32:41.059109 (+ 622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:32:41.059356 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:32:41.059360 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:32:41.059371 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:32:41.060025 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:41.060031 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:41.060750 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:41.060751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:41.061985 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:41.061987 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:41.062375 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:41.062376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:41.062528 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:41.063104 (+ 576us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:32:41.063104 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:32:41.063143 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:32:41.063171 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:32:41.063365 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:32:41.063370 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:32:41.063373 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:32:41.063442 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 06:32:41.063546 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":100,"threads_started":1}
I20260928 06:32:41.064397 10991 heartbeater.cc:347] Connected to a master server at 127.22.225.190:37703
I20260928 06:32:41.064515 10991 heartbeater.cc:464] Registering TS with master...
I20260928 06:32:41.064728 10991 heartbeater.cc:511] Master 127.22.225.190:37703 requested a full tablet report, sending...
I20260928 06:32:41.065147 10672 ts_manager.cc:194] Registered new tserver with Master: 84f00bd875e448bfb4dd02794539d1b2 (127.22.225.130:35279)
I20260928 06:32:41.065665 10672 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.225.130:33379
I20260928 06:32:41.066638 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 10863
I20260928 06:32:41.066769 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576899143097-23430-0/minicluster-data/ts-1/wal/instance
I20260928 06:32:41.066975 23430 external_mini_cluster.cc:1428] Setting key a74c7fd6a42d56975675ef3f4949f2f5
I20260928 06:32:41.067895 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576899143097-23430-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576899143097-23430-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576899143097-23430-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.131:0
--local_ip_for_outbound_sockets=127.22.225.131
--webserver_interface=127.22.225.131
--webserver_port=0
--tserver_master_addrs=127.22.225.190:37703
--builtin_ntp_servers=127.22.225.148:33587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 06:32:41.157480 10994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:32:41.157722 10994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:32:41.157770 10994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:32:41.158987 10994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:32:41.159057 10994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:32:41.159101 10994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.131
I20260928 06:32:41.160516 10994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:33587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576899143097-23430-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576899143097-23430-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.225.131
--webserver_port=0
--tserver_master_addrs=127.22.225.190:37703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10994
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.131
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576899143097-23430-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:32:41.160837 10994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:32:41.161162 10994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:32:41.167223 11002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:41.167209 11000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:41.167207 10999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:32:41.167728 10994 server_base.cc:1034] running on GCE node
I20260928 06:32:41.167981 10994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:32:41.168247 10994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:32:41.169402 10994 hybrid_clock.cc:648] HybridClock initialized: now 1790577161169386 us; error 34 us; skew 500 ppm
I20260928 06:32:41.170821 10994 webserver.cc:460] Webserver started at http://127.22.225.131:43945/ using document root <none> and password file <none>
I20260928 06:32:41.171103 10994 fs_manager.cc:359] Metadata directory not provided
I20260928 06:32:41.171175 10994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:32:41.171324 10994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:32:41.172410 10994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576899143097-23430-0/minicluster-data/ts-2/data/instance:
uuid: "dd3b6a4c287d41039e0f8a059a2cdb64"
format_stamp: "Formatted at 2026-09-28 06:32:41 on dist-test-slave-d59m"
server_key: "6f948059f4cfa4443e6c037bda6fad97"
server_key_iv: "476e04b2b27748442e73412738cccad0"
server_key_version: "encryptionkey@0"
I20260928 06:32:41.172816 10994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576899143097-23430-0/minicluster-data/ts-2/wal/instance:
uuid: "dd3b6a4c287d41039e0f8a059a2cdb64"
format_stamp: "Formatted at 2026-09-28 06:32:41 on dist-test-slave-d59m"
server_key: "6f948059f4cfa4443e6c037bda6fad97"
server_key_iv: "476e04b2b27748442e73412738cccad0"
server_key_version: "encryptionkey@0"
I20260928 06:32:41.174186 10994 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:32:41.174971 11008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:41.175127 10994 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:32:41.175230 10994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576899143097-23430-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576899143097-23430-0/minicluster-data/ts-2/wal
uuid: "dd3b6a4c287d41039e0f8a059a2cdb64"
format_stamp: "Formatted at 2026-09-28 06:32:41 on dist-test-slave-d59m"
server_key: "6f948059f4cfa4443e6c037bda6fad97"
server_key_iv: "476e04b2b27748442e73412738cccad0"
server_key_version: "encryptionkey@0"
I20260928 06:32:41.175304 10994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576899143097-23430-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576899143097-23430-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576899143097-23430-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:32:41.189847 10994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:32:41.190214 10994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:32:41.190380 10994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:32:41.190651 10994 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:32:41.191023 10994 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:32:41.191078 10994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:41.191123 10994 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:32:41.191149 10994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:41.198422 10994 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.131:41201
I20260928 06:32:41.198462 11121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.131:41201 every 8 connection(s)
I20260928 06:32:41.198812 10994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576899143097-23430-0/minicluster-data/ts-2/data/info.pb
I20260928 06:32:41.203506 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 10994
I20260928 06:32:41.203652 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576899143097-23430-0/minicluster-data/ts-2/wal/instance
I20260928 06:32:41.203894 23430 external_mini_cluster.cc:1428] Setting key 45beaa73dee58e6e14462951f04587bd
I20260928 06:32:41.205363 10711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:41.199625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:34615 (local address 127.22.225.190:37703)
0928 06:32:41.199863 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:41.199866 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:41.199956 (+ 90us) server_negotiation.cc:407] Connection header received
0928 06:32:41.200640 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:41.200642 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:41.200655 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:41.200719 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:32:41.201611 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:41.202094 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:41.203575 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:41.203890 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:41.203975 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:41.205071 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:32:41.205072 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:32:41.205072 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:32:41.205078 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:32:41.205093 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:32:41.205097 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:32:41.205153 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:41.205260 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 06:32:41.205287 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191}
I20260928 06:32:41.205591 11124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:41.199608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:37703 (local address 127.22.225.131:34615)
0928 06:32:41.199820 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:41.199869 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 06:32:41.200498 (+ 629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:32:41.200803 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:32:41.200808 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:32:41.200819 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:32:41.201489 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:41.201496 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:41.202181 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:41.202182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:41.203467 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:41.203470 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:41.204019 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:41.204020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:41.204183 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:41.204919 (+ 736us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:32:41.204920 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:32:41.204951 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:32:41.204974 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:32:41.205156 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:32:41.205161 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:32:41.205164 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:32:41.205257 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 06:32:41.205362 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":87,"threads_started":1}
I20260928 06:32:41.206096 11122 heartbeater.cc:347] Connected to a master server at 127.22.225.190:37703
I20260928 06:32:41.206200 11122 heartbeater.cc:464] Registering TS with master...
I20260928 06:32:41.206395 11122 heartbeater.cc:511] Master 127.22.225.190:37703 requested a full tablet report, sending...
I20260928 06:32:41.206740 10672 ts_manager.cc:194] Registered new tserver with Master: dd3b6a4c287d41039e0f8a059a2cdb64 (127.22.225.131:41201)
I20260928 06:32:41.207190 10672 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.225.131:34615
I20260928 06:32:41.216181 23430 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:32:41.221295 10711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:41.217867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45412 (local address 127.22.225.190:37703)
0928 06:32:41.217930 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:41.217932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:41.218078 (+ 146us) server_negotiation.cc:407] Connection header received
0928 06:32:41.218139 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:41.218139 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:41.218150 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:41.218200 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:32:41.218341 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:41.218827 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:41.219946 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:41.220130 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:41.220217 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:41.221037 (+ 820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:32:41.221038 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:32:41.221039 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:32:41.221044 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:32:41.221058 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:32:41.221062 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:32:41.221116 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:41.221176 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 06:32:41.221202 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 06:32:41.223146 10672 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45412:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:32:41.225310 10672 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:32:41.233202 11131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:41.227618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.130:35279 (local address 127.0.0.1:47794)
0928 06:32:41.228596 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:41.228607 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:32:41.228662 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:32:41.228868 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:32:41.228871 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:32:41.228874 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:32:41.228956 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:41.228958 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:41.230348 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:41.230350 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:41.231719 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:41.231722 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:41.232868 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:41.232869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:41.232947 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:41.232960 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 06:32:41.233033 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 06:32:41.233087 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":946,"thread_start_us":66,"threads_started":1}
I20260928 06:32:41.233330 11132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:41.227838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.129:36607 (local address 127.0.0.1:45442)
0928 06:32:41.228658 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:41.228667 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:32:41.228715 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:32:41.229056 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:32:41.229057 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:32:41.229060 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:32:41.229113 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:41.229115 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:41.230646 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:41.230647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:41.231823 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:41.231825 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:41.233118 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:41.233119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:41.233183 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:41.233188 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:32:41.233217 (+ 29us) client_negotiation.cc:240] Negotiation successful
0928 06:32:41.233255 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":808,"thread_start_us":48,"threads_started":1}
I20260928 06:32:41.233670 11130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:41.227602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.131:41201 (local address 127.0.0.1:33230)
0928 06:32:41.228362 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:41.228411 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 06:32:41.228472 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:32:41.229291 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:32:41.229292 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:32:41.229294 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:32:41.229346 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:41.229347 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:41.231025 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:41.231026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:41.232065 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:41.232067 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:41.233450 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:41.233451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:41.233521 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:41.233525 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:32:41.233565 (+ 40us) client_negotiation.cc:240] Negotiation successful
0928 06:32:41.233607 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":734,"thread_start_us":75,"threads_started":1}
I20260928 06:32:41.233749 11134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:41.227990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45442 (local address 127.22.225.129:36607)
0928 06:32:41.228199 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:41.228205 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:41.228905 (+ 700us) server_negotiation.cc:407] Connection header received
0928 06:32:41.228947 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:41.228949 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:41.228963 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:41.229177 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:32:41.229235 (+ 58us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:41.230565 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:41.231925 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:41.233030 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:41.233494 (+ 464us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:41.233591 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:41.233616 (+ 25us) server_negotiation.cc:299] Negotiation successful
0928 06:32:41.233653 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":100,"threads_started":1}
I20260928 06:32:41.234323 11135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:41.228014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33230 (local address 127.22.225.131:41201)
0928 06:32:41.228855 (+ 841us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:41.228859 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:41.228869 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:32:41.228905 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:41.228907 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:41.228921 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:41.229001 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:32:41.229469 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:41.230937 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:41.232397 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:41.233122 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:41.233182 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:41.234169 (+ 987us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:41.234198 (+ 29us) server_negotiation.cc:299] Negotiation successful
0928 06:32:41.234237 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":781,"thread_start_us":80,"threads_started":1}
I20260928 06:32:41.234541 10793 tablet_service.cc:1467] Processing CreateTablet for tablet 1b2681a8fca04f55a70b8be41db204c2 (DEFAULT_TABLE table=test-table [id=5bd7be92b13c49809bbed00353551697]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:32:41.234925 10793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b2681a8fca04f55a70b8be41db204c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:32:41.235047 11056 tablet_service.cc:1467] Processing CreateTablet for tablet 1b2681a8fca04f55a70b8be41db204c2 (DEFAULT_TABLE table=test-table [id=5bd7be92b13c49809bbed00353551697]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:32:41.235390 11056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b2681a8fca04f55a70b8be41db204c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:32:41.238058 11136 tablet_bootstrap.cc:492] T 1b2681a8fca04f55a70b8be41db204c2 P 5735acfb6e2a48deadf6b4a52bfe214c: Bootstrap starting.
I20260928 06:32:41.238339 11133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:41.227795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47794 (local address 127.22.225.130:35279)
0928 06:32:41.228299 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:41.228305 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:41.228665 (+ 360us) server_negotiation.cc:407] Connection header received
0928 06:32:41.228731 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:41.228733 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:41.228750 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:41.228838 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:32:41.229053 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:41.230261 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:41.231827 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:41.232667 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:41.232733 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:41.232833 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:41.233631 (+ 798us) server_negotiation.cc:299] Negotiation successful
0928 06:32:41.233671 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":105,"threads_started":1}
I20260928 06:32:41.238986 11136 tablet_bootstrap.cc:654] T 1b2681a8fca04f55a70b8be41db204c2 P 5735acfb6e2a48deadf6b4a52bfe214c: Neither blocks nor log segments found. Creating new log.
I20260928 06:32:41.239003 10925 tablet_service.cc:1467] Processing CreateTablet for tablet 1b2681a8fca04f55a70b8be41db204c2 (DEFAULT_TABLE table=test-table [id=5bd7be92b13c49809bbed00353551697]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:32:41.239334 11136 log.cc:824] T 1b2681a8fca04f55a70b8be41db204c2 P 5735acfb6e2a48deadf6b4a52bfe214c: Log is configured to *not* fsync() on all Append() calls
I20260928 06:32:41.239357 10925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b2681a8fca04f55a70b8be41db204c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:32:41.240146 11136 tablet_bootstrap.cc:492] T 1b2681a8fca04f55a70b8be41db204c2 P 5735acfb6e2a48deadf6b4a52bfe214c: No bootstrap required, opened a new log
I20260928 06:32:41.240222 11136 ts_tablet_manager.cc:1397] T 1b2681a8fca04f55a70b8be41db204c2 P 5735acfb6e2a48deadf6b4a52bfe214c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:32:41.242691 11139 tablet_bootstrap.cc:492] T 1b2681a8fca04f55a70b8be41db204c2 P 84f00bd875e448bfb4dd02794539d1b2: Bootstrap starting.
I20260928 06:32:41.242864 11136 raft_consensus.cc:348] T 1b2681a8fca04f55a70b8be41db204c2 P 5735acfb6e2a48deadf6b4a52bfe214c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd3b6a4c287d41039e0f8a059a2cdb64" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 41201 } } peers { permanent_uuid: "5735acfb6e2a48deadf6b4a52bfe214c" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 36607 } } peers { permanent_uuid: "84f00bd875e448bfb4dd02794539d1b2" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 35279 } }
I20260928 06:32:41.243050 11136 raft_consensus.cc:374] T 1b2681a8fca04f55a70b8be41db204c2 P 5735acfb6e2a48deadf6b4a52bfe214c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:32:41.243091 11136 raft_consensus.cc:729] T 1b2681a8fca04f55a70b8be41db204c2 P 5735acfb6e2a48deadf6b4a52bfe214c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5735acfb6e2a48deadf6b4a52bfe214c, State: Initialized, Role: FOLLOWER
I20260928 06:32:41.243239 11136 consensus_queue.cc:260] T 1b2681a8fca04f55a70b8be41db204c2 P 5735acfb6e2a48deadf6b4a52bfe214c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd3b6a4c287d41039e0f8a059a2cdb64" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 41201 } } peers { permanent_uuid: "5735acfb6e2a48deadf6b4a52bfe214c" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 36607 } } peers { permanent_uuid: "84f00bd875e448bfb4dd02794539d1b2" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 35279 } }
I20260928 06:32:41.243532 11136 ts_tablet_manager.cc:1428] T 1b2681a8fca04f55a70b8be41db204c2 P 5735acfb6e2a48deadf6b4a52bfe214c: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260928 06:32:41.243669 10860 heartbeater.cc:503] Master 127.22.225.190:37703 was elected leader, sending a full tablet report...
I20260928 06:32:41.243903 11139 tablet_bootstrap.cc:654] T 1b2681a8fca04f55a70b8be41db204c2 P 84f00bd875e448bfb4dd02794539d1b2: Neither blocks nor log segments found. Creating new log.
I20260928 06:32:41.244222 11137 tablet_bootstrap.cc:492] T 1b2681a8fca04f55a70b8be41db204c2 P dd3b6a4c287d41039e0f8a059a2cdb64: Bootstrap starting.
I20260928 06:32:41.244874 11139 log.cc:824] T 1b2681a8fca04f55a70b8be41db204c2 P 84f00bd875e448bfb4dd02794539d1b2: Log is configured to *not* fsync() on all Append() calls
I20260928 06:32:41.244971 11137 tablet_bootstrap.cc:654] T 1b2681a8fca04f55a70b8be41db204c2 P dd3b6a4c287d41039e0f8a059a2cdb64: Neither blocks nor log segments found. Creating new log.
I20260928 06:32:41.245247 11137 log.cc:824] T 1b2681a8fca04f55a70b8be41db204c2 P dd3b6a4c287d41039e0f8a059a2cdb64: Log is configured to *not* fsync() on all Append() calls
I20260928 06:32:41.245625 11139 tablet_bootstrap.cc:492] T 1b2681a8fca04f55a70b8be41db204c2 P 84f00bd875e448bfb4dd02794539d1b2: No bootstrap required, opened a new log
I20260928 06:32:41.245705 11139 ts_tablet_manager.cc:1397] T 1b2681a8fca04f55a70b8be41db204c2 P 84f00bd875e448bfb4dd02794539d1b2: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260928 06:32:41.246070 11137 tablet_bootstrap.cc:492] T 1b2681a8fca04f55a70b8be41db204c2 P dd3b6a4c287d41039e0f8a059a2cdb64: No bootstrap required, opened a new log
I20260928 06:32:41.246182 11137 ts_tablet_manager.cc:1397] T 1b2681a8fca04f55a70b8be41db204c2 P dd3b6a4c287d41039e0f8a059a2cdb64: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 06:32:41.248286 11137 raft_consensus.cc:348] T 1b2681a8fca04f55a70b8be41db204c2 P dd3b6a4c287d41039e0f8a059a2cdb64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd3b6a4c287d41039e0f8a059a2cdb64" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 41201 } } peers { permanent_uuid: "5735acfb6e2a48deadf6b4a52bfe214c" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 36607 } } peers { permanent_uuid: "84f00bd875e448bfb4dd02794539d1b2" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 35279 } }
I20260928 06:32:41.248471 11137 raft_consensus.cc:374] T 1b2681a8fca04f55a70b8be41db204c2 P dd3b6a4c287d41039e0f8a059a2cdb64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:32:41.248534 11137 raft_consensus.cc:729] T 1b2681a8fca04f55a70b8be41db204c2 P dd3b6a4c287d41039e0f8a059a2cdb64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd3b6a4c287d41039e0f8a059a2cdb64, State: Initialized, Role: FOLLOWER
I20260928 06:32:41.248601 11139 raft_consensus.cc:348] T 1b2681a8fca04f55a70b8be41db204c2 P 84f00bd875e448bfb4dd02794539d1b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd3b6a4c287d41039e0f8a059a2cdb64" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 41201 } } peers { permanent_uuid: "5735acfb6e2a48deadf6b4a52bfe214c" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 36607 } } peers { permanent_uuid: "84f00bd875e448bfb4dd02794539d1b2" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 35279 } }
I20260928 06:32:41.248703 11137 consensus_queue.cc:260] T 1b2681a8fca04f55a70b8be41db204c2 P dd3b6a4c287d41039e0f8a059a2cdb64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd3b6a4c287d41039e0f8a059a2cdb64" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 41201 } } peers { permanent_uuid: "5735acfb6e2a48deadf6b4a52bfe214c" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 36607 } } peers { permanent_uuid: "84f00bd875e448bfb4dd02794539d1b2" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 35279 } }
I20260928 06:32:41.248769 11139 raft_consensus.cc:374] T 1b2681a8fca04f55a70b8be41db204c2 P 84f00bd875e448bfb4dd02794539d1b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:32:41.248798 11139 raft_consensus.cc:729] T 1b2681a8fca04f55a70b8be41db204c2 P 84f00bd875e448bfb4dd02794539d1b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84f00bd875e448bfb4dd02794539d1b2, State: Initialized, Role: FOLLOWER
I20260928 06:32:41.248974 11137 ts_tablet_manager.cc:1428] T 1b2681a8fca04f55a70b8be41db204c2 P dd3b6a4c287d41039e0f8a059a2cdb64: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260928 06:32:41.248955 11139 consensus_queue.cc:260] T 1b2681a8fca04f55a70b8be41db204c2 P 84f00bd875e448bfb4dd02794539d1b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd3b6a4c287d41039e0f8a059a2cdb64" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 41201 } } peers { permanent_uuid: "5735acfb6e2a48deadf6b4a52bfe214c" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 36607 } } peers { permanent_uuid: "84f00bd875e448bfb4dd02794539d1b2" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 35279 } }
I20260928 06:32:41.249121 11122 heartbeater.cc:503] Master 127.22.225.190:37703 was elected leader, sending a full tablet report...
I20260928 06:32:41.249492 11139 ts_tablet_manager.cc:1428] T 1b2681a8fca04f55a70b8be41db204c2 P 84f00bd875e448bfb4dd02794539d1b2: Time spent starting tablet: real 0.004s user 0.002s sys 0.002s
I20260928 06:32:41.249552 10991 heartbeater.cc:503] Master 127.22.225.190:37703 was elected leader, sending a full tablet report...
W20260928 06:32:41.308099 10992 tablet.cc:2369] T 1b2681a8fca04f55a70b8be41db204c2 P 84f00bd875e448bfb4dd02794539d1b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:32:41.380294 11143 raft_consensus.cc:482] T 1b2681a8fca04f55a70b8be41db204c2 P dd3b6a4c287d41039e0f8a059a2cdb64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:32:41.380481 11143 raft_consensus.cc:504] T 1b2681a8fca04f55a70b8be41db204c2 P dd3b6a4c287d41039e0f8a059a2cdb64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd3b6a4c287d41039e0f8a059a2cdb64" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 41201 } } peers { permanent_uuid: "5735acfb6e2a48deadf6b4a52bfe214c" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 36607 } } peers { permanent_uuid: "84f00bd875e448bfb4dd02794539d1b2" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 35279 } }
I20260928 06:32:41.380961 11143 leader_election.cc:288] T 1b2681a8fca04f55a70b8be41db204c2 P dd3b6a4c287d41039e0f8a059a2cdb64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5735acfb6e2a48deadf6b4a52bfe214c (127.22.225.129:36607), 84f00bd875e448bfb4dd02794539d1b2 (127.22.225.130:35279)
I20260928 06:32:41.384707 11145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:41.381139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.129:36607 (local address 127.22.225.131:46429)
0928 06:32:41.381478 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:41.381489 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:32:41.381565 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:32:41.381749 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:32:41.381749 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:32:41.381751 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:32:41.381812 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:41.381814 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:41.382788 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:41.382789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:41.383580 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:41.383582 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:41.384388 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:41.384389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:41.384473 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:41.384478 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:32:41.384534 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 06:32:41.384606 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":141,"thread_start_us":150,"threads_started":1}
I20260928 06:32:41.384699 11134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:41.381218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:46429 (local address 127.22.225.129:36607)
0928 06:32:41.381292 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:41.381294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:41.381540 (+ 246us) server_negotiation.cc:407] Connection header received
0928 06:32:41.381643 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:41.381644 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:41.381656 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:41.381686 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:32:41.381926 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:41.382570 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:41.383679 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:41.384303 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:41.384386 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:41.384394 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:41.384560 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 06:32:41.384627 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 06:32:41.384909 11124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:41.381135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.130:35279 (local address 127.22.225.131:37701)
0928 06:32:41.381217 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:41.381230 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:32:41.381289 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:32:41.381664 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:32:41.381665 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:32:41.381667 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:32:41.381738 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:41.381742 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:41.382713 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:41.382714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:41.383895 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:41.383898 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:41.384640 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:41.384641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:41.384715 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:41.384719 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:32:41.384764 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 06:32:41.384822 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260928 06:32:41.385200 10814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b2681a8fca04f55a70b8be41db204c2" candidate_uuid: "dd3b6a4c287d41039e0f8a059a2cdb64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5735acfb6e2a48deadf6b4a52bfe214c" is_pre_election: true
I20260928 06:32:41.385380 10814 raft_consensus.cc:2454] T 1b2681a8fca04f55a70b8be41db204c2 P 5735acfb6e2a48deadf6b4a52bfe214c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd3b6a4c287d41039e0f8a059a2cdb64 in term 0.
I20260928 06:32:41.385649 11012 leader_election.cc:302] T 1b2681a8fca04f55a70b8be41db204c2 P dd3b6a4c287d41039e0f8a059a2cdb64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5735acfb6e2a48deadf6b4a52bfe214c, dd3b6a4c287d41039e0f8a059a2cdb64; no voters:
I20260928 06:32:41.385689 11133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:41.381433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.225.131:37701 (local address 127.22.225.130:35279)
0928 06:32:41.381488 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:41.381490 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:41.381516 (+ 26us) server_negotiation.cc:407] Connection header received
0928 06:32:41.381546 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:41.381547 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:41.381561 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:41.381613 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:32:41.381837 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:41.382592 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:41.383987 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:41.384508 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:41.385535 (+ 1027us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:41.385546 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:41.385570 (+ 24us) server_negotiation.cc:299] Negotiation successful
0928 06:32:41.385616 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 06:32:41.385798 11143 raft_consensus.cc:2781] T 1b2681a8fca04f55a70b8be41db204c2 P dd3b6a4c287d41039e0f8a059a2cdb64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:32:41.385855 11143 raft_consensus.cc:482] T 1b2681a8fca04f55a70b8be41db204c2 P dd3b6a4c287d41039e0f8a059a2cdb64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:32:41.385892 11143 raft_consensus.cc:3037] T 1b2681a8fca04f55a70b8be41db204c2 P dd3b6a4c287d41039e0f8a059a2cdb64 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:32:41.385948 10945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b2681a8fca04f55a70b8be41db204c2" candidate_uuid: "dd3b6a4c287d41039e0f8a059a2cdb64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84f00bd875e448bfb4dd02794539d1b2" is_pre_election: true
I20260928 06:32:41.386087 10945 raft_consensus.cc:2454] T 1b2681a8fca04f55a70b8be41db204c2 P 84f00bd875e448bfb4dd02794539d1b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd3b6a4c287d41039e0f8a059a2cdb64 in term 0.
I20260928 06:32:41.386782 11143 raft_consensus.cc:504] T 1b2681a8fca04f55a70b8be41db204c2 P dd3b6a4c287d41039e0f8a059a2cdb64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd3b6a4c287d41039e0f8a059a2cdb64" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 41201 } } peers { permanent_uuid: "5735acfb6e2a48deadf6b4a52bfe214c" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 36607 } } peers { permanent_uuid: "84f00bd875e448bfb4dd02794539d1b2" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 35279 } }
I20260928 06:32:41.386940 11143 leader_election.cc:288] T 1b2681a8fca04f55a70b8be41db204c2 P dd3b6a4c287d41039e0f8a059a2cdb64 [CANDIDATE]: Term 1 election: Requested vote from peers 5735acfb6e2a48deadf6b4a52bfe214c (127.22.225.129:36607), 84f00bd875e448bfb4dd02794539d1b2 (127.22.225.130:35279)
I20260928 06:32:41.387177 10814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b2681a8fca04f55a70b8be41db204c2" candidate_uuid: "dd3b6a4c287d41039e0f8a059a2cdb64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5735acfb6e2a48deadf6b4a52bfe214c"
I20260928 06:32:41.387205 10945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b2681a8fca04f55a70b8be41db204c2" candidate_uuid: "dd3b6a4c287d41039e0f8a059a2cdb64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84f00bd875e448bfb4dd02794539d1b2"
I20260928 06:32:41.387272 10945 raft_consensus.cc:3037] T 1b2681a8fca04f55a70b8be41db204c2 P 84f00bd875e448bfb4dd02794539d1b2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:32:41.387265 10814 raft_consensus.cc:3037] T 1b2681a8fca04f55a70b8be41db204c2 P 5735acfb6e2a48deadf6b4a52bfe214c [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:32:41.388018 10945 raft_consensus.cc:2454] T 1b2681a8fca04f55a70b8be41db204c2 P 84f00bd875e448bfb4dd02794539d1b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd3b6a4c287d41039e0f8a059a2cdb64 in term 1.
I20260928 06:32:41.388209 10814 raft_consensus.cc:2454] T 1b2681a8fca04f55a70b8be41db204c2 P 5735acfb6e2a48deadf6b4a52bfe214c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd3b6a4c287d41039e0f8a059a2cdb64 in term 1.
I20260928 06:32:41.388227 11009 leader_election.cc:302] T 1b2681a8fca04f55a70b8be41db204c2 P dd3b6a4c287d41039e0f8a059a2cdb64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84f00bd875e448bfb4dd02794539d1b2, dd3b6a4c287d41039e0f8a059a2cdb64; no voters:
I20260928 06:32:41.388329 11143 raft_consensus.cc:2781] T 1b2681a8fca04f55a70b8be41db204c2 P dd3b6a4c287d41039e0f8a059a2cdb64 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:32:41.388477 11143 raft_consensus.cc:686] T 1b2681a8fca04f55a70b8be41db204c2 P dd3b6a4c287d41039e0f8a059a2cdb64 [term 1 LEADER]: Becoming Leader. State: Replica: dd3b6a4c287d41039e0f8a059a2cdb64, State: Running, Role: LEADER
I20260928 06:32:41.388561 11143 consensus_queue.cc:237] T 1b2681a8fca04f55a70b8be41db204c2 P dd3b6a4c287d41039e0f8a059a2cdb64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd3b6a4c287d41039e0f8a059a2cdb64" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 41201 } } peers { permanent_uuid: "5735acfb6e2a48deadf6b4a52bfe214c" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 36607 } } peers { permanent_uuid: "84f00bd875e448bfb4dd02794539d1b2" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 35279 } }
I20260928 06:32:41.389511 10672 catalog_manager.cc:5495] T 1b2681a8fca04f55a70b8be41db204c2 P dd3b6a4c287d41039e0f8a059a2cdb64 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd3b6a4c287d41039e0f8a059a2cdb64 (127.22.225.131). New cstate: current_term: 1 leader_uuid: "dd3b6a4c287d41039e0f8a059a2cdb64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd3b6a4c287d41039e0f8a059a2cdb64" member_type: VOTER last_known_addr { host: "127.22.225.131" port: 41201 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5735acfb6e2a48deadf6b4a52bfe214c" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 36607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84f00bd875e448bfb4dd02794539d1b2" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 35279 } health_report { overall_health: UNKNOWN } } }
W20260928 06:32:41.393438 10861 tablet.cc:2369] T 1b2681a8fca04f55a70b8be41db204c2 P 5735acfb6e2a48deadf6b4a52bfe214c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:32:41.416175 11135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:41.413549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33242 (local address 127.22.225.131:41201)
0928 06:32:41.413613 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:41.413616 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:41.413626 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:32:41.413744 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:41.413745 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:41.413764 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:41.413839 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:32:41.413980 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:41.414475 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:41.415317 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:41.415516 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:41.415593 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:41.415750 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:32:41.415875 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:32:41.415986 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:41.416005 (+ 19us) server_negotiation.cc:299] Negotiation successful
0928 06:32:41.416026 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 06:32:41.418370 10945 raft_consensus.cc:1261] T 1b2681a8fca04f55a70b8be41db204c2 P 84f00bd875e448bfb4dd02794539d1b2 [term 1 FOLLOWER]: Refusing update from remote peer dd3b6a4c287d41039e0f8a059a2cdb64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:32:41.418740 10814 raft_consensus.cc:1261] T 1b2681a8fca04f55a70b8be41db204c2 P 5735acfb6e2a48deadf6b4a52bfe214c [term 1 FOLLOWER]: Refusing update from remote peer dd3b6a4c287d41039e0f8a059a2cdb64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:32:41.418781 11146 consensus_queue.cc:1035] T 1b2681a8fca04f55a70b8be41db204c2 P dd3b6a4c287d41039e0f8a059a2cdb64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84f00bd875e448bfb4dd02794539d1b2" member_type: VOTER last_known_addr { host: "127.22.225.130" port: 35279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:32:41.419049 11146 consensus_queue.cc:1035] T 1b2681a8fca04f55a70b8be41db204c2 P dd3b6a4c287d41039e0f8a059a2cdb64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5735acfb6e2a48deadf6b4a52bfe214c" member_type: VOTER last_known_addr { host: "127.22.225.129" port: 36607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:32:41.423306 11149 mvcc.cc:205] Tried to move back new op lower bound from 7334204053166612480 to 7334204053048340480. Current Snapshot: MvccSnapshot[applied={T|T < 7334204053166612480 or (T in {7334204053166612480})}]
I20260928 06:32:41.424430 10672 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:32:41.424538 10672 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:32:41.425349 10672 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c96fa69ee475490689ac820066d8c9ca: Sending DeleteTablet for 3 replicas of tablet 1b2681a8fca04f55a70b8be41db204c2
I20260928 06:32:41.425805 10925 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b2681a8fca04f55a70b8be41db204c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:32:41 UTC) from {username='slave'} at 127.0.0.1:47794
I20260928 06:32:41.425886 11056 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b2681a8fca04f55a70b8be41db204c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:32:41 UTC) from {username='slave'} at 127.0.0.1:33230
I20260928 06:32:41.426110 10793 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b2681a8fca04f55a70b8be41db204c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:32:41 UTC) from {username='slave'} at 127.0.0.1:45442
I20260928 06:32:41.426344 11161 tablet_replica.cc:330] T 1b2681a8fca04f55a70b8be41db204c2 P 5735acfb6e2a48deadf6b4a52bfe214c: stopping tablet replica
I20260928 06:32:41.426484 11161 raft_consensus.cc:2229] T 1b2681a8fca04f55a70b8be41db204c2 P 5735acfb6e2a48deadf6b4a52bfe214c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:32:41.426570 11160 tablet_replica.cc:330] T 1b2681a8fca04f55a70b8be41db204c2 P dd3b6a4c287d41039e0f8a059a2cdb64: stopping tablet replica
I20260928 06:32:41.426640 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 10732
I20260928 06:32:41.426676 11160 raft_consensus.cc:2229] T 1b2681a8fca04f55a70b8be41db204c2 P dd3b6a4c287d41039e0f8a059a2cdb64 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:32:41.426678 11161 raft_consensus.cc:2258] T 1b2681a8fca04f55a70b8be41db204c2 P 5735acfb6e2a48deadf6b4a52bfe214c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:32:41.427206 11160 raft_consensus.cc:2258] T 1b2681a8fca04f55a70b8be41db204c2 P dd3b6a4c287d41039e0f8a059a2cdb64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:32:41.427727 11160 ts_tablet_manager.cc:1905] T 1b2681a8fca04f55a70b8be41db204c2 P dd3b6a4c287d41039e0f8a059a2cdb64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:32:41.429808 11160 ts_tablet_manager.cc:1918] T 1b2681a8fca04f55a70b8be41db204c2 P dd3b6a4c287d41039e0f8a059a2cdb64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:32:41.429888 11160 log.cc:1196] T 1b2681a8fca04f55a70b8be41db204c2 P dd3b6a4c287d41039e0f8a059a2cdb64: Deleting WAL directory at /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576899143097-23430-0/minicluster-data/ts-2/wal/wals/1b2681a8fca04f55a70b8be41db204c2
I20260928 06:32:41.430146 11160 ts_tablet_manager.cc:1939] T 1b2681a8fca04f55a70b8be41db204c2 P dd3b6a4c287d41039e0f8a059a2cdb64: Deleting consensus metadata
I20260928 06:32:41.430393 11159 tablet_replica.cc:330] T 1b2681a8fca04f55a70b8be41db204c2 P 84f00bd875e448bfb4dd02794539d1b2: stopping tablet replica
I20260928 06:32:41.430496 11159 raft_consensus.cc:2229] T 1b2681a8fca04f55a70b8be41db204c2 P 84f00bd875e448bfb4dd02794539d1b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:32:41.430577 11159 raft_consensus.cc:2258] T 1b2681a8fca04f55a70b8be41db204c2 P 84f00bd875e448bfb4dd02794539d1b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:32:41.431025 10657 catalog_manager.cc:4841] TS dd3b6a4c287d41039e0f8a059a2cdb64 (127.22.225.131:41201): tablet 1b2681a8fca04f55a70b8be41db204c2 (table test-table [id=5bd7be92b13c49809bbed00353551697]) successfully deleted
I20260928 06:32:41.431111 11159 ts_tablet_manager.cc:1905] T 1b2681a8fca04f55a70b8be41db204c2 P 84f00bd875e448bfb4dd02794539d1b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:32:41.432428 11159 ts_tablet_manager.cc:1918] T 1b2681a8fca04f55a70b8be41db204c2 P 84f00bd875e448bfb4dd02794539d1b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:32:41.432484 11159 log.cc:1196] T 1b2681a8fca04f55a70b8be41db204c2 P 84f00bd875e448bfb4dd02794539d1b2: Deleting WAL directory at /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576899143097-23430-0/minicluster-data/ts-1/wal/wals/1b2681a8fca04f55a70b8be41db204c2
I20260928 06:32:41.432754 11159 ts_tablet_manager.cc:1939] T 1b2681a8fca04f55a70b8be41db204c2 P 84f00bd875e448bfb4dd02794539d1b2: Deleting consensus metadata
I20260928 06:32:41.433202 10656 catalog_manager.cc:4841] TS 84f00bd875e448bfb4dd02794539d1b2 (127.22.225.130:35279): tablet 1b2681a8fca04f55a70b8be41db204c2 (table test-table [id=5bd7be92b13c49809bbed00353551697]) successfully deleted
W20260928 06:32:41.433316 10659 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.225.129:36607 (error 108)
I20260928 06:32:41.433948 11131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:41.433710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.129:36607 (local address 127.0.0.1:45448)
0928 06:32:41.433856 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:41.433900 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.129:36607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
W20260928 06:32:41.434063 10659 catalog_manager.cc:4584] TS 5735acfb6e2a48deadf6b4a52bfe214c (127.22.225.129:36607): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1b2681a8fca04f55a70b8be41db204c2: Network error: Client connection negotiation failed: client connection to 127.22.225.129:36607: connect: Connection refused (error 111)
I20260928 06:32:41.436287 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 10863
I20260928 06:32:41.442320 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 10994
I20260928 06:32:41.448201 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 10641
2026-09-28T06:32:41Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (892 ms)
[----------] 6 tests from EncryptionPolicyTest (9608 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790576899143097-23430-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:32:41 dist-test-slave-d59m krb5kdc[11165](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:32:41 dist-test-slave-d59m krb5kdc[11165](info): set up 2 sockets
Sep 28 06:32:41 dist-test-slave-d59m krb5kdc[11165](info): commencing operation
krb5kdc: starting...
W20260928 06:32:42.549098 23430 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Sep 28 06:32:42 dist-test-slave-d59m krb5kdc[11165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:32:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:32:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790576899143097-23430-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790576899143097-23430-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:32:42.685458 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790576899143097-23430-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790576899143097-23430-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790576899143097-23430-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40645
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790576899143097-23430-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40645
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:32:42.779166 11178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:32:42.779379 11178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:32:42.779402 11178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:32:42.780652 11178 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:32:42.780694 11178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:32:42.780709 11178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:32:42.780721 11178 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:32:42.780738 11178 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:32:42.782248 11178 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790576899143097-23430-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790576899143097-23430-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40645
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790576899143097-23430-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790576899143097-23430-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:32:42.782506 11178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:32:42.782752 11178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:32:42.787657 11184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:42.787667 11186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:32:42.787982 11178 server_base.cc:1034] running on GCE node
W20260928 06:32:42.787669 11183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:32:42.788270 11178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:32:42.788642 11178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:32:42.789850 11178 hybrid_clock.cc:648] HybridClock initialized: now 1790577162789825 us; error 47 us; skew 500 ppm
Sep 28 06:32:42 dist-test-slave-d59m krb5kdc[11165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:32:42.792629 11178 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:32:42.793155 11178 webserver.cc:460] Webserver started at http://127.0.0.1:33381/ using document root <none> and password file <none>
I20260928 06:32:42.793407 11178 fs_manager.cc:359] Metadata directory not provided
I20260928 06:32:42.793478 11178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:32:42.793640 11178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:32:42.794747 11178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790576899143097-23430-0/minicluster-data/master-0/data/instance:
uuid: "d3642bf3e9ec4a169a6cf3b0173a8e95"
format_stamp: "Formatted at 2026-09-28 06:32:42 on dist-test-slave-d59m"
server_key: "754db662ba4c8c4a62712ef4a28335d2"
server_key_iv: "89b2d911bbdab7f1a30517718c6b00be"
server_key_version: "encryptionkey@0"
I20260928 06:32:42.795128 11178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790576899143097-23430-0/minicluster-data/master-0/wal/instance:
uuid: "d3642bf3e9ec4a169a6cf3b0173a8e95"
format_stamp: "Formatted at 2026-09-28 06:32:42 on dist-test-slave-d59m"
server_key: "754db662ba4c8c4a62712ef4a28335d2"
server_key_iv: "89b2d911bbdab7f1a30517718c6b00be"
server_key_version: "encryptionkey@0"
I20260928 06:32:42.796574 11178 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 06:32:42.797525 11193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:42.797742 11178 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:32:42.797852 11178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790576899143097-23430-0/minicluster-data/master-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790576899143097-23430-0/minicluster-data/master-0/wal
uuid: "d3642bf3e9ec4a169a6cf3b0173a8e95"
format_stamp: "Formatted at 2026-09-28 06:32:42 on dist-test-slave-d59m"
server_key: "754db662ba4c8c4a62712ef4a28335d2"
server_key_iv: "89b2d911bbdab7f1a30517718c6b00be"
server_key_version: "encryptionkey@0"
I20260928 06:32:42.797940 11178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790576899143097-23430-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790576899143097-23430-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790576899143097-23430-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:32:42.812351 11178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:32:42.814930 11178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:32:42.815124 11178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:32:42.819705 11178 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40645
I20260928 06:32:42.819761 11245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40645 every 8 connection(s)
I20260928 06:32:42.820132 11178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
I20260928 06:32:42.821179 11246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:32:42.821372 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 11178
I20260928 06:32:42.821532 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790576899143097-23430-0/minicluster-data/master-0/wal/instance
I20260928 06:32:42.821799 23430 external_mini_cluster.cc:1428] Setting key 5f679c489066a660485b04de88a91ff8
I20260928 06:32:42.824222 11246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3642bf3e9ec4a169a6cf3b0173a8e95: Bootstrap starting.
I20260928 06:32:42.824976 11246 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3642bf3e9ec4a169a6cf3b0173a8e95: Neither blocks nor log segments found. Creating new log.
I20260928 06:32:42.825275 11246 log.cc:824] T 00000000000000000000000000000000 P d3642bf3e9ec4a169a6cf3b0173a8e95: Log is configured to *not* fsync() on all Append() calls
I20260928 06:32:42.826083 11246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3642bf3e9ec4a169a6cf3b0173a8e95: No bootstrap required, opened a new log
Sep 28 06:32:42 dist-test-slave-d59m krb5kdc[11165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 06:32:42.828617 11246 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3642bf3e9ec4a169a6cf3b0173a8e95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3642bf3e9ec4a169a6cf3b0173a8e95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40645 } }
I20260928 06:32:42.828830 11246 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3642bf3e9ec4a169a6cf3b0173a8e95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:32:42.828881 11246 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3642bf3e9ec4a169a6cf3b0173a8e95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3642bf3e9ec4a169a6cf3b0173a8e95, State: Initialized, Role: FOLLOWER
I20260928 06:32:42.829006 11246 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3642bf3e9ec4a169a6cf3b0173a8e95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3642bf3e9ec4a169a6cf3b0173a8e95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40645 } }
I20260928 06:32:42.829084 11246 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3642bf3e9ec4a169a6cf3b0173a8e95 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:32:42.829114 11246 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3642bf3e9ec4a169a6cf3b0173a8e95 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:32:42.829180 11246 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3642bf3e9ec4a169a6cf3b0173a8e95 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:32:42.829903 11246 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3642bf3e9ec4a169a6cf3b0173a8e95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3642bf3e9ec4a169a6cf3b0173a8e95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40645 } }
I20260928 06:32:42.830008 11246 leader_election.cc:302] T 00000000000000000000000000000000 P d3642bf3e9ec4a169a6cf3b0173a8e95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3642bf3e9ec4a169a6cf3b0173a8e95; no voters:
I20260928 06:32:42.830171 11246 leader_election.cc:288] T 00000000000000000000000000000000 P d3642bf3e9ec4a169a6cf3b0173a8e95 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:32:42.830226 11251 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3642bf3e9ec4a169a6cf3b0173a8e95 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:32:42.830526 11246 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3642bf3e9ec4a169a6cf3b0173a8e95 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:32:42.830755 11251 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3642bf3e9ec4a169a6cf3b0173a8e95 [term 1 LEADER]: Becoming Leader. State: Replica: d3642bf3e9ec4a169a6cf3b0173a8e95, State: Running, Role: LEADER
I20260928 06:32:42.830899 11251 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3642bf3e9ec4a169a6cf3b0173a8e95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3642bf3e9ec4a169a6cf3b0173a8e95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40645 } }
I20260928 06:32:42.831293 11253 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3642bf3e9ec4a169a6cf3b0173a8e95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3642bf3e9ec4a169a6cf3b0173a8e95. Latest consensus state: current_term: 1 leader_uuid: "d3642bf3e9ec4a169a6cf3b0173a8e95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3642bf3e9ec4a169a6cf3b0173a8e95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40645 } } }
I20260928 06:32:42.832000 11253 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3642bf3e9ec4a169a6cf3b0173a8e95 [sys.catalog]: This master's current role is: LEADER
I20260928 06:32:42.831650 11252 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3642bf3e9ec4a169a6cf3b0173a8e95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3642bf3e9ec4a169a6cf3b0173a8e95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3642bf3e9ec4a169a6cf3b0173a8e95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40645 } } }
I20260928 06:32:42.832290 11252 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3642bf3e9ec4a169a6cf3b0173a8e95 [sys.catalog]: This master's current role is: LEADER
W20260928 06:32:42.832746 11266 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d3642bf3e9ec4a169a6cf3b0173a8e95: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:32:42.832803 11266 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:32:42.832849 11267 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:32:42.833004 11267 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:32:42.833542 11249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:42.822405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44671 (local address 127.0.0.1:40645)
0928 06:32:42.822565 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:42.822577 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:42.822660 (+ 83us) server_negotiation.cc:407] Connection header received
0928 06:32:42.823143 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:42.823151 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:42.823207 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:42.823363 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:32:42.824022 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:42.825107 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:42.826138 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:42.826565 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:42.826690 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:42.829352 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:32:42.829357 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:32:42.829358 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:32:42.829371 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:32:42.831603 (+ 2232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:32:42.832125 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:32:42.832128 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:32:42.832129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:32:42.832256 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:32:42.832548 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:32:42.832549 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:32:42.832549 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:32:42.832933 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:32:42.833096 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:42.833214 (+ 118us) server_negotiation.cc:299] Negotiation successful
0928 06:32:42.833318 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":68,"threads_started":1}
I20260928 06:32:42.834522 23430 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:32:42.836995 11267 catalog_manager.cc:1347] Generated new cluster ID: f5c0776d94e14d6380cb534a780e4c41
I20260928 06:32:42.837052 11267 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:32:42.850116 11249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:42.835881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58431 (local address 127.0.0.1:40645)
0928 06:32:42.835924 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:42.835926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:42.837315 (+ 1389us) server_negotiation.cc:407] Connection header received
0928 06:32:42.837392 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:42.837393 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:42.837404 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:42.837461 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:32:42.837983 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:42.838550 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:42.839280 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:42.839523 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:42.839598 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:42.840927 (+ 1329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:32:42.840928 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:32:42.840928 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:32:42.840936 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:32:42.845381 (+ 4445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:32:42.849120 (+ 3739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:32:42.849122 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:32:42.849122 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:32:42.849192 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:32:42.849451 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:32:42.849452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:32:42.849452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:32:42.849583 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:32:42.849674 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:42.849936 (+ 262us) server_negotiation.cc:299] Negotiation successful
0928 06:32:42.850013 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:32:42.851060 11267 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:32:42.851743 11267 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:32:42.859292 11267 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3642bf3e9ec4a169a6cf3b0173a8e95: Generated new TSK 0
I20260928 06:32:42.859491 11267 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:32:42.863945 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 11178
W20260928 06:32:42.869267 11163 connection.cc:535] client connection to 127.0.0.1:40645 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40645): Connection reset by peer (error 104)
2026-09-28T06:32:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1413 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790576899143097-23430-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:32:42 dist-test-slave-d59m krb5kdc[11277](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:32:42 dist-test-slave-d59m krb5kdc[11277](info): set up 2 sockets
Sep 28 06:32:42 dist-test-slave-d59m krb5kdc[11277](info): commencing operation
krb5kdc: starting...
W20260928 06:32:43.909130 23430 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Sep 28 06:32:43 dist-test-slave-d59m krb5kdc[11277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:32:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:32:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790576899143097-23430-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790576899143097-23430-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:32:44.043453 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790576899143097-23430-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790576899143097-23430-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790576899143097-23430-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46651
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790576899143097-23430-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46651
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:32:44.162896 11290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:32:44.163203 11290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:32:44.163241 11290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:32:44.164815 11290 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:32:44.164885 11290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:32:44.164904 11290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:32:44.164922 11290 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:32:44.164937 11290 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:32:44.166939 11290 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790576899143097-23430-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790576899143097-23430-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46651
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790576899143097-23430-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790576899143097-23430-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:32:44.167207 11290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:32:44.167461 11290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:32:44.172892 11295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:44.172890 11296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:32:44.173120 11290 server_base.cc:1034] running on GCE node
W20260928 06:32:44.173348 11298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:32:44.173599 11290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:32:44.173946 11290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:32:44.175123 11290 hybrid_clock.cc:648] HybridClock initialized: now 1790577164175100 us; error 36 us; skew 500 ppm
Sep 28 06:32:44 dist-test-slave-d59m krb5kdc[11277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577164, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:32:44.182009 11290 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:32:44.182571 11290 webserver.cc:460] Webserver started at http://127.0.0.1:40113/ using document root <none> and password file <none>
I20260928 06:32:44.182819 11290 fs_manager.cc:359] Metadata directory not provided
I20260928 06:32:44.182886 11290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:32:44.183046 11290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:32:44.184213 11290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790576899143097-23430-0/minicluster-data/master-0/data/instance:
uuid: "9c366d46134049ab8a4766df9987a792"
format_stamp: "Formatted at 2026-09-28 06:32:44 on dist-test-slave-d59m"
server_key: "1f70c6358875d5c09549a22d87ad80ff"
server_key_iv: "649a79f6d5531799d158d66efedf4b22"
server_key_version: "encryptionkey@0"
I20260928 06:32:44.184648 11290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790576899143097-23430-0/minicluster-data/master-0/wal/instance:
uuid: "9c366d46134049ab8a4766df9987a792"
format_stamp: "Formatted at 2026-09-28 06:32:44 on dist-test-slave-d59m"
server_key: "1f70c6358875d5c09549a22d87ad80ff"
server_key_iv: "649a79f6d5531799d158d66efedf4b22"
server_key_version: "encryptionkey@0"
I20260928 06:32:44.186174 11290 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:32:44.187081 11305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:44.187320 11290 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:32:44.187433 11290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790576899143097-23430-0/minicluster-data/master-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790576899143097-23430-0/minicluster-data/master-0/wal
uuid: "9c366d46134049ab8a4766df9987a792"
format_stamp: "Formatted at 2026-09-28 06:32:44 on dist-test-slave-d59m"
server_key: "1f70c6358875d5c09549a22d87ad80ff"
server_key_iv: "649a79f6d5531799d158d66efedf4b22"
server_key_version: "encryptionkey@0"
I20260928 06:32:44.187525 11290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790576899143097-23430-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790576899143097-23430-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790576899143097-23430-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:32:44.203241 11290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:32:44.205550 11290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:32:44.205727 11290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:32:44.209977 11290 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46651
I20260928 06:32:44.210033 11357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46651 every 8 connection(s)
I20260928 06:32:44.210446 11290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
I20260928 06:32:44.211513 11358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:32:44.214527 11358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c366d46134049ab8a4766df9987a792: Bootstrap starting.
I20260928 06:32:44.215217 11358 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c366d46134049ab8a4766df9987a792: Neither blocks nor log segments found. Creating new log.
I20260928 06:32:44.215552 11358 log.cc:824] T 00000000000000000000000000000000 P 9c366d46134049ab8a4766df9987a792: Log is configured to *not* fsync() on all Append() calls
I20260928 06:32:44.216365 11358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c366d46134049ab8a4766df9987a792: No bootstrap required, opened a new log
I20260928 06:32:44.218477 11358 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c366d46134049ab8a4766df9987a792 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c366d46134049ab8a4766df9987a792" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46651 } }
I20260928 06:32:44.218665 11358 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c366d46134049ab8a4766df9987a792 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:32:44.218703 11358 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c366d46134049ab8a4766df9987a792 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c366d46134049ab8a4766df9987a792, State: Initialized, Role: FOLLOWER
I20260928 06:32:44.218847 11358 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c366d46134049ab8a4766df9987a792 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c366d46134049ab8a4766df9987a792" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46651 } }
I20260928 06:32:44.218920 11358 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c366d46134049ab8a4766df9987a792 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:32:44.218947 11358 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c366d46134049ab8a4766df9987a792 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:32:44.218991 11358 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c366d46134049ab8a4766df9987a792 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:32:44.219492 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 11290
I20260928 06:32:44.219627 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790576899143097-23430-0/minicluster-data/master-0/wal/instance
I20260928 06:32:44.219693 11358 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c366d46134049ab8a4766df9987a792 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c366d46134049ab8a4766df9987a792" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46651 } }
I20260928 06:32:44.219802 11358 leader_election.cc:302] T 00000000000000000000000000000000 P 9c366d46134049ab8a4766df9987a792 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c366d46134049ab8a4766df9987a792; no voters:
I20260928 06:32:44.219879 23430 external_mini_cluster.cc:1428] Setting key 355aec1fa25fffeabf638807ad87aad5
I20260928 06:32:44.219969 11358 leader_election.cc:288] T 00000000000000000000000000000000 P 9c366d46134049ab8a4766df9987a792 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:32:44.220010 11361 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c366d46134049ab8a4766df9987a792 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:32:44.220234 11361 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c366d46134049ab8a4766df9987a792 [term 1 LEADER]: Becoming Leader. State: Replica: 9c366d46134049ab8a4766df9987a792, State: Running, Role: LEADER
I20260928 06:32:44.220407 11361 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c366d46134049ab8a4766df9987a792 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c366d46134049ab8a4766df9987a792" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46651 } }
I20260928 06:32:44.220445 11358 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c366d46134049ab8a4766df9987a792 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:32:44.221536 11362 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c366d46134049ab8a4766df9987a792 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c366d46134049ab8a4766df9987a792. Latest consensus state: current_term: 1 leader_uuid: "9c366d46134049ab8a4766df9987a792" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c366d46134049ab8a4766df9987a792" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46651 } } }
I20260928 06:32:44.221614 11362 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c366d46134049ab8a4766df9987a792 [sys.catalog]: This master's current role is: LEADER
I20260928 06:32:44.221938 11363 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c366d46134049ab8a4766df9987a792 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c366d46134049ab8a4766df9987a792" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c366d46134049ab8a4766df9987a792" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46651 } } }
I20260928 06:32:44.222141 11363 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c366d46134049ab8a4766df9987a792 [sys.catalog]: This master's current role is: LEADER
I20260928 06:32:44.222302 11374 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:32:44.222795 11374 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:32:44.225561 11374 catalog_manager.cc:1347] Generated new cluster ID: c2bc217bb604410e92a74290d35130c9
I20260928 06:32:44.225610 11374 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:32:44.226424 11365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:44.220546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53767 (local address 127.0.0.1:46651)
0928 06:32:44.220747 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:44.220758 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:44.220803 (+ 45us) server_negotiation.cc:407] Connection header received
0928 06:32:44.220966 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:44.220975 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:44.221042 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:44.221475 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:32:44.221599 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:44.222897 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:44.224010 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:44.224603 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:44.225060 (+ 457us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:44.225556 (+ 496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:32:44.225561 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:32:44.225561 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:32:44.225765 (+ 204us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:32:44.225803 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:32:44.225830 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:32:44.226134 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:44.226178 (+ 44us) server_negotiation.cc:299] Negotiation successful
0928 06:32:44.226207 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":87,"threads_started":1}
I20260928 06:32:44.227370 23430 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:32:44.233089 11365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:44.228950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50697 (local address 127.0.0.1:46651)
0928 06:32:44.228983 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:44.228985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:44.229246 (+ 261us) server_negotiation.cc:407] Connection header received
0928 06:32:44.229279 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:44.229280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:44.229290 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:44.229349 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:32:44.229539 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:44.230309 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:44.231432 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:44.231749 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:44.231790 (+ 41us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:44.232706 (+ 916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:32:44.232707 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:32:44.232708 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:32:44.232713 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:32:44.232729 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:32:44.232734 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:32:44.232768 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:44.232984 (+ 216us) server_negotiation.cc:299] Negotiation successful
0928 06:32:44.233010 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 06:32:44.237777 11374 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:32:44.238396 11374 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:32:44.242853 11374 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c366d46134049ab8a4766df9987a792: Generated new TSK 0
I20260928 06:32:44.243027 11374 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:32:44.249563 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 11290
W20260928 06:32:44.254563 11275 connection.cc:535] client connection to 127.0.0.1:46651 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46651): Connection reset by peer (error 104)
2026-09-28T06:32:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1385 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790576899143097-23430-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:32:44 dist-test-slave-d59m krb5kdc[11389](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:32:44 dist-test-slave-d59m krb5kdc[11389](info): set up 2 sockets
Sep 28 06:32:44 dist-test-slave-d59m krb5kdc[11389](info): commencing operation
krb5kdc: starting...
W20260928 06:32:45.313153 23430 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Sep 28 06:32:45 dist-test-slave-d59m krb5kdc[11389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:32:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:32:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790576899143097-23430-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790576899143097-23430-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:32:45.457525 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790576899143097-23430-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790576899143097-23430-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790576899143097-23430-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38231
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790576899143097-23430-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38231
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:32:45.544160 11402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:32:45.544381 11402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:32:45.544406 11402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:32:45.545650 11402 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:32:45.545697 11402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:32:45.545711 11402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:32:45.545727 11402 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:32:45.545745 11402 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:32:45.547326 11402 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790576899143097-23430-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790576899143097-23430-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38231
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790576899143097-23430-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11402
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790576899143097-23430-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:32:45.547602 11402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:32:45.547842 11402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:32:45.553207 11407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:32:45.553330 11402 server_base.cc:1034] running on GCE node
W20260928 06:32:45.553455 11408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:45.553503 11410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:32:45.553858 11402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:32:45.554252 11402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:32:45.555434 11402 hybrid_clock.cc:648] HybridClock initialized: now 1790577165555413 us; error 45 us; skew 500 ppm
Sep 28 06:32:45 dist-test-slave-d59m krb5kdc[11389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:32:45.558590 11402 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:32:45.559079 11402 webserver.cc:460] Webserver started at http://127.0.0.1:45639/ using document root <none> and password file <none>
I20260928 06:32:45.559331 11402 fs_manager.cc:359] Metadata directory not provided
I20260928 06:32:45.559402 11402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:32:45.559551 11402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:32:45.560711 11402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790576899143097-23430-0/minicluster-data/master-0/data/instance:
uuid: "f89d90d1c2cf4da081db00e9b84aea11"
format_stamp: "Formatted at 2026-09-28 06:32:45 on dist-test-slave-d59m"
server_key: "e64cc52a33a8de4fdf4472d3691a685e"
server_key_iv: "bb7302f87c168572305fb21ef55bb745"
server_key_version: "encryptionkey@0"
I20260928 06:32:45.561091 11402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790576899143097-23430-0/minicluster-data/master-0/wal/instance:
uuid: "f89d90d1c2cf4da081db00e9b84aea11"
format_stamp: "Formatted at 2026-09-28 06:32:45 on dist-test-slave-d59m"
server_key: "e64cc52a33a8de4fdf4472d3691a685e"
server_key_iv: "bb7302f87c168572305fb21ef55bb745"
server_key_version: "encryptionkey@0"
I20260928 06:32:45.562515 11402 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:32:45.563336 11417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:45.563565 11402 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:32:45.563686 11402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790576899143097-23430-0/minicluster-data/master-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790576899143097-23430-0/minicluster-data/master-0/wal
uuid: "f89d90d1c2cf4da081db00e9b84aea11"
format_stamp: "Formatted at 2026-09-28 06:32:45 on dist-test-slave-d59m"
server_key: "e64cc52a33a8de4fdf4472d3691a685e"
server_key_iv: "bb7302f87c168572305fb21ef55bb745"
server_key_version: "encryptionkey@0"
I20260928 06:32:45.563771 11402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790576899143097-23430-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790576899143097-23430-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790576899143097-23430-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:32:45.567142 11402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:32:45.569624 11402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:32:45.569777 11402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:32:45.573984 11402 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38231
I20260928 06:32:45.574030 11469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38231 every 8 connection(s)
I20260928 06:32:45.574375 11402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
I20260928 06:32:45.575686 11470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:32:45.579100 11470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f89d90d1c2cf4da081db00e9b84aea11: Bootstrap starting.
I20260928 06:32:45.579872 11470 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f89d90d1c2cf4da081db00e9b84aea11: Neither blocks nor log segments found. Creating new log.
I20260928 06:32:45.580207 11470 log.cc:824] T 00000000000000000000000000000000 P f89d90d1c2cf4da081db00e9b84aea11: Log is configured to *not* fsync() on all Append() calls
I20260928 06:32:45.581110 11470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f89d90d1c2cf4da081db00e9b84aea11: No bootstrap required, opened a new log
I20260928 06:32:45.582639 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 11402
I20260928 06:32:45.582778 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790576899143097-23430-0/minicluster-data/master-0/wal/instance
I20260928 06:32:45.582971 23430 external_mini_cluster.cc:1428] Setting key cc66ef001982f465f56e58f943304274
I20260928 06:32:45.583802 11470 raft_consensus.cc:348] T 00000000000000000000000000000000 P f89d90d1c2cf4da081db00e9b84aea11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f89d90d1c2cf4da081db00e9b84aea11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38231 } }
I20260928 06:32:45.584082 11470 raft_consensus.cc:374] T 00000000000000000000000000000000 P f89d90d1c2cf4da081db00e9b84aea11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:32:45.584175 11470 raft_consensus.cc:729] T 00000000000000000000000000000000 P f89d90d1c2cf4da081db00e9b84aea11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f89d90d1c2cf4da081db00e9b84aea11, State: Initialized, Role: FOLLOWER
I20260928 06:32:45.584795 11470 consensus_queue.cc:260] T 00000000000000000000000000000000 P f89d90d1c2cf4da081db00e9b84aea11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f89d90d1c2cf4da081db00e9b84aea11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38231 } }
I20260928 06:32:45.584998 11470 raft_consensus.cc:388] T 00000000000000000000000000000000 P f89d90d1c2cf4da081db00e9b84aea11 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:32:45.585083 11470 raft_consensus.cc:482] T 00000000000000000000000000000000 P f89d90d1c2cf4da081db00e9b84aea11 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:32:45.585145 11470 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f89d90d1c2cf4da081db00e9b84aea11 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:32:45.585474 11474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:45.583533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41589 (local address 127.0.0.1:38231)
0928 06:32:45.583869 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:45.583877 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:45.583902 (+ 25us) server_negotiation.cc:407] Connection header received
0928 06:32:45.584001 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:45.584007 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:45.584029 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:45.584188 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:32:45.584925 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:32:45.584929 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:32:45.584929 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:32:45.584933 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:32:45.584981 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:32:45.584993 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:32:45.585070 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:45.585222 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 06:32:45.585316 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":88,"threads_started":1}
I20260928 06:32:45.585990 11470 raft_consensus.cc:504] T 00000000000000000000000000000000 P f89d90d1c2cf4da081db00e9b84aea11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f89d90d1c2cf4da081db00e9b84aea11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38231 } }
I20260928 06:32:45.586148 11470 leader_election.cc:302] T 00000000000000000000000000000000 P f89d90d1c2cf4da081db00e9b84aea11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f89d90d1c2cf4da081db00e9b84aea11; no voters:
I20260928 06:32:45.586344 11470 leader_election.cc:288] T 00000000000000000000000000000000 P f89d90d1c2cf4da081db00e9b84aea11 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:32:45.586340 23430 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:32:45.586527 11476 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f89d90d1c2cf4da081db00e9b84aea11 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:32:45.586625 11470 sys_catalog.cc:564] T 00000000000000000000000000000000 P f89d90d1c2cf4da081db00e9b84aea11 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:32:45.586700 11476 raft_consensus.cc:686] T 00000000000000000000000000000000 P f89d90d1c2cf4da081db00e9b84aea11 [term 1 LEADER]: Becoming Leader. State: Replica: f89d90d1c2cf4da081db00e9b84aea11, State: Running, Role: LEADER
I20260928 06:32:45.586826 11476 consensus_queue.cc:237] T 00000000000000000000000000000000 P f89d90d1c2cf4da081db00e9b84aea11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f89d90d1c2cf4da081db00e9b84aea11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38231 } }
I20260928 06:32:45.587162 11476 sys_catalog.cc:455] T 00000000000000000000000000000000 P f89d90d1c2cf4da081db00e9b84aea11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f89d90d1c2cf4da081db00e9b84aea11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f89d90d1c2cf4da081db00e9b84aea11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38231 } } }
I20260928 06:32:45.587222 11476 sys_catalog.cc:458] T 00000000000000000000000000000000 P f89d90d1c2cf4da081db00e9b84aea11 [sys.catalog]: This master's current role is: LEADER
I20260928 06:32:45.587355 11476 sys_catalog.cc:455] T 00000000000000000000000000000000 P f89d90d1c2cf4da081db00e9b84aea11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f89d90d1c2cf4da081db00e9b84aea11. Latest consensus state: current_term: 1 leader_uuid: "f89d90d1c2cf4da081db00e9b84aea11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f89d90d1c2cf4da081db00e9b84aea11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38231 } } }
I20260928 06:32:45.587410 11476 sys_catalog.cc:458] T 00000000000000000000000000000000 P f89d90d1c2cf4da081db00e9b84aea11 [sys.catalog]: This master's current role is: LEADER
I20260928 06:32:45.587672 11481 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:32:45.588351 11481 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:32:45.589398 11474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:45.587903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60699 (local address 127.0.0.1:38231)
0928 06:32:45.587945 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:45.587947 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:45.588221 (+ 274us) server_negotiation.cc:407] Connection header received
0928 06:32:45.588275 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:45.588276 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:45.588282 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:45.588367 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:32:45.589135 (+ 768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:32:45.589136 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:32:45.589137 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:32:45.589141 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:32:45.589157 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:32:45.589163 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:32:45.589199 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:45.589321 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 06:32:45.589337 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:32:45.590261 11481 catalog_manager.cc:1347] Generated new cluster ID: 6ebcf911bb9540e2a9027ce8086261bf
I20260928 06:32:45.590417 11481 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:32:45.612962 11481 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:32:45.613153 11481 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:32:45.615509 11481 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f89d90d1c2cf4da081db00e9b84aea11: Generated new TSK 0
I20260928 06:32:45.615696 11481 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:32:45.623454 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 11402
2026-09-28T06:32:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1374 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790576899143097-23430-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:32:45 dist-test-slave-d59m krb5kdc[11501](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:32:45 dist-test-slave-d59m krb5kdc[11501](info): set up 2 sockets
Sep 28 06:32:45 dist-test-slave-d59m krb5kdc[11501](info): commencing operation
krb5kdc: starting...
W20260928 06:32:46.794750 23430 mini_kdc.cc:121] Time spent starting KDC: real 1.151s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Sep 28 06:32:46 dist-test-slave-d59m krb5kdc[11501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:32:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:32:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790576899143097-23430-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790576899143097-23430-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:32:46.929606 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790576899143097-23430-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790576899143097-23430-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790576899143097-23430-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38567
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790576899143097-23430-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38567
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:32:47.047960 11514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:32:47.048180 11514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:32:47.048206 11514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:32:47.049541 11514 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:32:47.049595 11514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:32:47.049610 11514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:32:47.049623 11514 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:32:47.049635 11514 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:32:47.051223 11514 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790576899143097-23430-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790576899143097-23430-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38567
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790576899143097-23430-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790576899143097-23430-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:32:47.051510 11514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:32:47.051745 11514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:32:47.056967 11519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:47.057087 11520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:32:47.057000 11514 server_base.cc:1034] running on GCE node
W20260928 06:32:47.056962 11522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:32:47.057482 11514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:32:47.057776 11514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:32:47.058961 11514 hybrid_clock.cc:648] HybridClock initialized: now 1790577167058946 us; error 32 us; skew 500 ppm
Sep 28 06:32:47 dist-test-slave-d59m krb5kdc[11501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577167, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:32:47.062098 11514 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:32:47.062723 11514 webserver.cc:460] Webserver started at http://127.0.0.1:45841/ using document root <none> and password file <none>
I20260928 06:32:47.063046 11514 fs_manager.cc:359] Metadata directory not provided
I20260928 06:32:47.063122 11514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:32:47.063290 11514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:32:47.065008 11514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790576899143097-23430-0/minicluster-data/master-0/data/instance:
uuid: "ceb6964bba794948a38e61abeef1fe8a"
format_stamp: "Formatted at 2026-09-28 06:32:47 on dist-test-slave-d59m"
server_key: "b1ee64192e8200b4aa7796198e6db119"
server_key_iv: "d7592d23e609fd2f36ee3d90f845e4e0"
server_key_version: "encryptionkey@0"
I20260928 06:32:47.065474 11514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790576899143097-23430-0/minicluster-data/master-0/wal/instance:
uuid: "ceb6964bba794948a38e61abeef1fe8a"
format_stamp: "Formatted at 2026-09-28 06:32:47 on dist-test-slave-d59m"
server_key: "b1ee64192e8200b4aa7796198e6db119"
server_key_iv: "d7592d23e609fd2f36ee3d90f845e4e0"
server_key_version: "encryptionkey@0"
I20260928 06:32:47.067416 11514 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:32:47.068428 11529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:47.068634 11514 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:32:47.068742 11514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790576899143097-23430-0/minicluster-data/master-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790576899143097-23430-0/minicluster-data/master-0/wal
uuid: "ceb6964bba794948a38e61abeef1fe8a"
format_stamp: "Formatted at 2026-09-28 06:32:47 on dist-test-slave-d59m"
server_key: "b1ee64192e8200b4aa7796198e6db119"
server_key_iv: "d7592d23e609fd2f36ee3d90f845e4e0"
server_key_version: "encryptionkey@0"
I20260928 06:32:47.068841 11514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790576899143097-23430-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790576899143097-23430-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790576899143097-23430-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:32:47.082336 11514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:32:47.085608 11514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:32:47.085857 11514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:32:47.091399 11514 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38567
I20260928 06:32:47.091449 11581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38567 every 8 connection(s)
I20260928 06:32:47.092283 11514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
I20260928 06:32:47.093237 11582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:32:47.095932 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 11514
I20260928 06:32:47.096057 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790576899143097-23430-0/minicluster-data/master-0/wal/instance
I20260928 06:32:47.096279 23430 external_mini_cluster.cc:1428] Setting key 9bc44e3304a82a9e805dbc33a4479b33
I20260928 06:32:47.097764 11582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ceb6964bba794948a38e61abeef1fe8a: Bootstrap starting.
I20260928 06:32:47.098436 11582 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ceb6964bba794948a38e61abeef1fe8a: Neither blocks nor log segments found. Creating new log.
I20260928 06:32:47.098774 11582 log.cc:824] T 00000000000000000000000000000000 P ceb6964bba794948a38e61abeef1fe8a: Log is configured to *not* fsync() on all Append() calls
I20260928 06:32:47.099649 11582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ceb6964bba794948a38e61abeef1fe8a: No bootstrap required, opened a new log
I20260928 06:32:47.102209 11582 raft_consensus.cc:348] T 00000000000000000000000000000000 P ceb6964bba794948a38e61abeef1fe8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ceb6964bba794948a38e61abeef1fe8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38567 } }
I20260928 06:32:47.102376 11582 raft_consensus.cc:374] T 00000000000000000000000000000000 P ceb6964bba794948a38e61abeef1fe8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:32:47.102442 11582 raft_consensus.cc:729] T 00000000000000000000000000000000 P ceb6964bba794948a38e61abeef1fe8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceb6964bba794948a38e61abeef1fe8a, State: Initialized, Role: FOLLOWER
I20260928 06:32:47.102581 11582 consensus_queue.cc:260] T 00000000000000000000000000000000 P ceb6964bba794948a38e61abeef1fe8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ceb6964bba794948a38e61abeef1fe8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38567 } }
I20260928 06:32:47.102676 11582 raft_consensus.cc:388] T 00000000000000000000000000000000 P ceb6964bba794948a38e61abeef1fe8a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:32:47.102733 11582 raft_consensus.cc:482] T 00000000000000000000000000000000 P ceb6964bba794948a38e61abeef1fe8a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:32:47.102785 11582 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ceb6964bba794948a38e61abeef1fe8a [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:32:47.103473 11582 raft_consensus.cc:504] T 00000000000000000000000000000000 P ceb6964bba794948a38e61abeef1fe8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ceb6964bba794948a38e61abeef1fe8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38567 } }
I20260928 06:32:47.103587 11582 leader_election.cc:302] T 00000000000000000000000000000000 P ceb6964bba794948a38e61abeef1fe8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ceb6964bba794948a38e61abeef1fe8a; no voters:
I20260928 06:32:47.103844 11582 leader_election.cc:288] T 00000000000000000000000000000000 P ceb6964bba794948a38e61abeef1fe8a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:32:47.103850 11587 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ceb6964bba794948a38e61abeef1fe8a [term 1 FOLLOWER]: Leader election won for term 1
Sep 28 06:32:47 dist-test-slave-d59m krb5kdc[11501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 06:32:47.104172 11582 sys_catalog.cc:564] T 00000000000000000000000000000000 P ceb6964bba794948a38e61abeef1fe8a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:32:47.104265 11587 raft_consensus.cc:686] T 00000000000000000000000000000000 P ceb6964bba794948a38e61abeef1fe8a [term 1 LEADER]: Becoming Leader. State: Replica: ceb6964bba794948a38e61abeef1fe8a, State: Running, Role: LEADER
I20260928 06:32:47.104408 11587 consensus_queue.cc:237] T 00000000000000000000000000000000 P ceb6964bba794948a38e61abeef1fe8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ceb6964bba794948a38e61abeef1fe8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38567 } }
I20260928 06:32:47.104820 11588 sys_catalog.cc:455] T 00000000000000000000000000000000 P ceb6964bba794948a38e61abeef1fe8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ceb6964bba794948a38e61abeef1fe8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ceb6964bba794948a38e61abeef1fe8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38567 } } }
I20260928 06:32:47.104936 11588 sys_catalog.cc:458] T 00000000000000000000000000000000 P ceb6964bba794948a38e61abeef1fe8a [sys.catalog]: This master's current role is: LEADER
I20260928 06:32:47.104882 11589 sys_catalog.cc:455] T 00000000000000000000000000000000 P ceb6964bba794948a38e61abeef1fe8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ceb6964bba794948a38e61abeef1fe8a. Latest consensus state: current_term: 1 leader_uuid: "ceb6964bba794948a38e61abeef1fe8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ceb6964bba794948a38e61abeef1fe8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38567 } } }
I20260928 06:32:47.105019 11589 sys_catalog.cc:458] T 00000000000000000000000000000000 P ceb6964bba794948a38e61abeef1fe8a [sys.catalog]: This master's current role is: LEADER
W20260928 06:32:47.106670 11603 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ceb6964bba794948a38e61abeef1fe8a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:32:47.106746 11603 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:32:47.106814 11594 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:32:47.106993 11594 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:32:47.109613 11585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:47.096849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35533 (local address 127.0.0.1:38567)
0928 06:32:47.097023 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:47.097033 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:47.097067 (+ 34us) server_negotiation.cc:407] Connection header received
0928 06:32:47.097635 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:47.097644 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:47.097708 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:47.097877 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:32:47.098650 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:47.100022 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:47.100848 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:47.101483 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:47.101615 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:47.105341 (+ 3726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:32:47.105345 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:32:47.105345 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:32:47.105404 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:32:47.107643 (+ 2239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:32:47.108129 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:32:47.108132 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:32:47.108132 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:32:47.108304 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:32:47.108543 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:32:47.108544 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:32:47.108545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:32:47.108875 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:32:47.109112 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:47.109188 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 06:32:47.109353 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":89,"threads_started":1}
I20260928 06:32:47.110352 11594 catalog_manager.cc:1347] Generated new cluster ID: de2d621ce46e4bff85ca26b5351caaed
I20260928 06:32:47.110437 11594 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:32:47.110555 23430 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:32:47.120357 11585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:47.112132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59789 (local address 127.0.0.1:38567)
0928 06:32:47.112167 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:47.112168 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:47.112232 (+ 64us) server_negotiation.cc:407] Connection header received
0928 06:32:47.112303 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:47.112304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:47.112314 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:47.112377 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:32:47.113133 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:47.114000 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:47.115040 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:47.115531 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:47.115596 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:47.117138 (+ 1542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:32:47.117140 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:32:47.117140 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:32:47.117148 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:32:47.119022 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:32:47.119412 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:32:47.119413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:32:47.119414 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:32:47.119464 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:32:47.119626 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:32:47.119626 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:32:47.119627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:32:47.119742 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:32:47.119823 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:47.120174 (+ 351us) server_negotiation.cc:299] Negotiation successful
0928 06:32:47.120247 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 06:32:47.124709 11594 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:32:47.125628 11594 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:32:47.137009 11594 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ceb6964bba794948a38e61abeef1fe8a: Generated new TSK 0
I20260928 06:32:47.137290 11594 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:32:47.142629 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 11514
2026-09-28T06:32:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1518 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790576899143097-23430-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:32:47 dist-test-slave-d59m krb5kdc[11613](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:32:47 dist-test-slave-d59m krb5kdc[11613](info): set up 2 sockets
Sep 28 06:32:47 dist-test-slave-d59m krb5kdc[11613](info): commencing operation
krb5kdc: starting...
W20260928 06:32:48.261435 23430 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Sep 28 06:32:48 dist-test-slave-d59m krb5kdc[11613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:32:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:32:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790576899143097-23430-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790576899143097-23430-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:32:48.403661 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790576899143097-23430-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790576899143097-23430-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790576899143097-23430-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39023
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790576899143097-23430-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39023
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:32:48.497509 11626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:32:48.497748 11626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:32:48.497787 11626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:32:48.499714 11626 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:32:48.499781 11626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:32:48.499805 11626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:32:48.499838 11626 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:32:48.499864 11626 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:32:48.501652 11626 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790576899143097-23430-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790576899143097-23430-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790576899143097-23430-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790576899143097-23430-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:32:48.501945 11626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:32:48.502168 11626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:32:48.507117 11634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:48.507103 11632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:32:48.507325 11626 server_base.cc:1034] running on GCE node
W20260928 06:32:48.507104 11631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:32:48.507766 11626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:32:48.508091 11626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:32:48.509264 11626 hybrid_clock.cc:648] HybridClock initialized: now 1790577168509251 us; error 35 us; skew 500 ppm
Sep 28 06:32:48 dist-test-slave-d59m krb5kdc[11613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:32:48.512090 11626 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:32:48.512677 11626 webserver.cc:460] Webserver started at http://127.0.0.1:43721/ using document root <none> and password file <none>
I20260928 06:32:48.512959 11626 fs_manager.cc:359] Metadata directory not provided
I20260928 06:32:48.513041 11626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:32:48.513186 11626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:32:48.514672 11626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790576899143097-23430-0/minicluster-data/master-0/data/instance:
uuid: "7e188551a6a846df9ba9d0f66f6ceb71"
format_stamp: "Formatted at 2026-09-28 06:32:48 on dist-test-slave-d59m"
server_key: "e7ef17cb92fb1d4140f99888ec5e127d"
server_key_iv: "4362125280e4aa3cdfbd45ddd5d1f085"
server_key_version: "encryptionkey@0"
I20260928 06:32:48.515127 11626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790576899143097-23430-0/minicluster-data/master-0/wal/instance:
uuid: "7e188551a6a846df9ba9d0f66f6ceb71"
format_stamp: "Formatted at 2026-09-28 06:32:48 on dist-test-slave-d59m"
server_key: "e7ef17cb92fb1d4140f99888ec5e127d"
server_key_iv: "4362125280e4aa3cdfbd45ddd5d1f085"
server_key_version: "encryptionkey@0"
I20260928 06:32:48.516952 11626 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260928 06:32:48.517854 11641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:48.518074 11626 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:32:48.518175 11626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790576899143097-23430-0/minicluster-data/master-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790576899143097-23430-0/minicluster-data/master-0/wal
uuid: "7e188551a6a846df9ba9d0f66f6ceb71"
format_stamp: "Formatted at 2026-09-28 06:32:48 on dist-test-slave-d59m"
server_key: "e7ef17cb92fb1d4140f99888ec5e127d"
server_key_iv: "4362125280e4aa3cdfbd45ddd5d1f085"
server_key_version: "encryptionkey@0"
I20260928 06:32:48.518257 11626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790576899143097-23430-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790576899143097-23430-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790576899143097-23430-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:32:48.534124 11626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:32:48.536003 11626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:32:48.536120 11626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:32:48.540156 11626 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39023
I20260928 06:32:48.540216 11693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39023 every 8 connection(s)
I20260928 06:32:48.540561 11626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
I20260928 06:32:48.541476 11694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:32:48.543744 11694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e188551a6a846df9ba9d0f66f6ceb71: Bootstrap starting.
I20260928 06:32:48.544283 11694 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e188551a6a846df9ba9d0f66f6ceb71: Neither blocks nor log segments found. Creating new log.
I20260928 06:32:48.544543 11694 log.cc:824] T 00000000000000000000000000000000 P 7e188551a6a846df9ba9d0f66f6ceb71: Log is configured to *not* fsync() on all Append() calls
I20260928 06:32:48.545280 11694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e188551a6a846df9ba9d0f66f6ceb71: No bootstrap required, opened a new log
I20260928 06:32:48.546880 11694 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e188551a6a846df9ba9d0f66f6ceb71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e188551a6a846df9ba9d0f66f6ceb71" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39023 } }
I20260928 06:32:48.547022 11694 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e188551a6a846df9ba9d0f66f6ceb71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:32:48.547045 11694 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e188551a6a846df9ba9d0f66f6ceb71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e188551a6a846df9ba9d0f66f6ceb71, State: Initialized, Role: FOLLOWER
I20260928 06:32:48.547159 11694 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e188551a6a846df9ba9d0f66f6ceb71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e188551a6a846df9ba9d0f66f6ceb71" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39023 } }
I20260928 06:32:48.547209 11694 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e188551a6a846df9ba9d0f66f6ceb71 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:32:48.547230 11694 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e188551a6a846df9ba9d0f66f6ceb71 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:32:48.547257 11694 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e188551a6a846df9ba9d0f66f6ceb71 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:32:48.547770 11694 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e188551a6a846df9ba9d0f66f6ceb71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e188551a6a846df9ba9d0f66f6ceb71" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39023 } }
I20260928 06:32:48.547859 11694 leader_election.cc:302] T 00000000000000000000000000000000 P 7e188551a6a846df9ba9d0f66f6ceb71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e188551a6a846df9ba9d0f66f6ceb71; no voters:
I20260928 06:32:48.547997 11694 leader_election.cc:288] T 00000000000000000000000000000000 P 7e188551a6a846df9ba9d0f66f6ceb71 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:32:48.548063 11697 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e188551a6a846df9ba9d0f66f6ceb71 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:32:48.548243 11697 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e188551a6a846df9ba9d0f66f6ceb71 [term 1 LEADER]: Becoming Leader. State: Replica: 7e188551a6a846df9ba9d0f66f6ceb71, State: Running, Role: LEADER
I20260928 06:32:48.548264 11694 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e188551a6a846df9ba9d0f66f6ceb71 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:32:48.548336 11697 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e188551a6a846df9ba9d0f66f6ceb71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e188551a6a846df9ba9d0f66f6ceb71" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39023 } }
I20260928 06:32:48.548753 11699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e188551a6a846df9ba9d0f66f6ceb71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e188551a6a846df9ba9d0f66f6ceb71. Latest consensus state: current_term: 1 leader_uuid: "7e188551a6a846df9ba9d0f66f6ceb71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e188551a6a846df9ba9d0f66f6ceb71" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39023 } } }
I20260928 06:32:48.548834 11699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e188551a6a846df9ba9d0f66f6ceb71 [sys.catalog]: This master's current role is: LEADER
I20260928 06:32:48.548931 11698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e188551a6a846df9ba9d0f66f6ceb71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e188551a6a846df9ba9d0f66f6ceb71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e188551a6a846df9ba9d0f66f6ceb71" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39023 } } }
I20260928 06:32:48.549001 11698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e188551a6a846df9ba9d0f66f6ceb71 [sys.catalog]: This master's current role is: LEADER
I20260928 06:32:48.549228 11706 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:32:48.549746 11706 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:32:48.550034 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 11626
I20260928 06:32:48.550184 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790576899143097-23430-0/minicluster-data/master-0/wal/instance
I20260928 06:32:48.550381 23430 external_mini_cluster.cc:1428] Setting key cdc53de1b8d1376b6ad3b2a2c6743857
I20260928 06:32:48.552672 11713 mvcc.cc:205] Tried to move back new op lower bound from 7334204082383896576 to 7334204082374344704. Current Snapshot: MvccSnapshot[applied={T|T < 7334204082383896576}]
I20260928 06:32:48.557272 11706 catalog_manager.cc:1347] Generated new cluster ID: 3e1f971f015049e7b05c163a951de184
I20260928 06:32:48.557339 11706 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:32:48.562837 11714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:48.550818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34907 (local address 127.0.0.1:39023)
0928 06:32:48.556858 (+ 6040us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:48.556868 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:48.557591 (+ 723us) server_negotiation.cc:407] Connection header received
0928 06:32:48.557736 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:48.557743 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:48.557799 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:48.557945 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:32:48.558193 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:48.559421 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:48.560503 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:48.561174 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:48.561294 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:48.562039 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:32:48.562045 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:32:48.562046 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:32:48.562315 (+ 269us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:32:48.562366 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:32:48.562382 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:32:48.562473 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:48.562543 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 06:32:48.562610 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5987,"thread_start_us":89,"threads_started":1}
I20260928 06:32:48.563795 23430 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:32:48.567032 11706 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:32:48.567536 11706 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:32:48.572269 11714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:48.565314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59903 (local address 127.0.0.1:39023)
0928 06:32:48.565349 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:48.565350 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:48.568784 (+ 3434us) server_negotiation.cc:407] Connection header received
0928 06:32:48.568892 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:48.568894 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:48.568906 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:48.568965 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:32:48.569177 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:48.569941 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:48.570952 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:48.571265 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:48.571331 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:48.572030 (+ 699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:32:48.572032 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:32:48.572032 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:32:48.572039 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:32:48.572054 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:32:48.572058 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:32:48.572108 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:48.572168 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 06:32:48.572194 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:32:48.574818 11706 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e188551a6a846df9ba9d0f66f6ceb71: Generated new TSK 0
I20260928 06:32:48.574985 11706 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:32:48.585307 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 11626
2026-09-28T06:32:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1447 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790576899143097-23430-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:32:48 dist-test-slave-d59m krb5kdc[11725](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:32:48 dist-test-slave-d59m krb5kdc[11725](info): set up 2 sockets
Sep 28 06:32:48 dist-test-slave-d59m krb5kdc[11725](info): commencing operation
krb5kdc: starting...
W20260928 06:32:49.745642 23430 mini_kdc.cc:121] Time spent starting KDC: real 1.136s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Sep 28 06:32:49 dist-test-slave-d59m krb5kdc[11725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:32:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:32:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790576899143097-23430-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790576899143097-23430-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:32:49.870922 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790576899143097-23430-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790576899143097-23430-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790576899143097-23430-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40081
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790576899143097-23430-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40081
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:32:49.962492 11738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:32:49.962742 11738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:32:49.962770 11738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:32:49.963953 11738 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:32:49.964016 11738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:32:49.964041 11738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:32:49.964068 11738 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:32:49.964092 11738 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:32:49.965813 11738 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790576899143097-23430-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790576899143097-23430-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40081
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790576899143097-23430-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40081
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790576899143097-23430-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:32:49.966112 11738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:32:49.966341 11738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:32:49.972028 11743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:49.972154 11746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:49.972028 11744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:32:49.972046 11738 server_base.cc:1034] running on GCE node
I20260928 06:32:49.972456 11738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:32:49.972820 11738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:32:49.973996 11738 hybrid_clock.cc:648] HybridClock initialized: now 1790577169973975 us; error 35 us; skew 500 ppm
Sep 28 06:32:49 dist-test-slave-d59m krb5kdc[11725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:32:49.982702 11738 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:32:49.983165 11738 webserver.cc:460] Webserver started at http://127.0.0.1:44511/ using document root <none> and password file <none>
I20260928 06:32:49.983421 11738 fs_manager.cc:359] Metadata directory not provided
I20260928 06:32:49.983490 11738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:32:49.983633 11738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:32:49.984848 11738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790576899143097-23430-0/minicluster-data/master-0/data/instance:
uuid: "b922dae8c7ab46c89fba21146626e98a"
format_stamp: "Formatted at 2026-09-28 06:32:49 on dist-test-slave-d59m"
server_key: "eda2f9033488d730194590c6524803b9"
server_key_iv: "79e1b2fa4a41a580157fdcf472f64b79"
server_key_version: "encryptionkey@0"
I20260928 06:32:49.985208 11738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790576899143097-23430-0/minicluster-data/master-0/wal/instance:
uuid: "b922dae8c7ab46c89fba21146626e98a"
format_stamp: "Formatted at 2026-09-28 06:32:49 on dist-test-slave-d59m"
server_key: "eda2f9033488d730194590c6524803b9"
server_key_iv: "79e1b2fa4a41a580157fdcf472f64b79"
server_key_version: "encryptionkey@0"
I20260928 06:32:49.987016 11738 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:32:49.987905 11753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:49.988076 11738 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:32:49.988170 11738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790576899143097-23430-0/minicluster-data/master-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790576899143097-23430-0/minicluster-data/master-0/wal
uuid: "b922dae8c7ab46c89fba21146626e98a"
format_stamp: "Formatted at 2026-09-28 06:32:49 on dist-test-slave-d59m"
server_key: "eda2f9033488d730194590c6524803b9"
server_key_iv: "79e1b2fa4a41a580157fdcf472f64b79"
server_key_version: "encryptionkey@0"
I20260928 06:32:49.988237 11738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790576899143097-23430-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790576899143097-23430-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790576899143097-23430-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:32:49.992709 11738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:32:49.995097 11738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:32:49.995256 11738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:32:49.999540 11738 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40081
I20260928 06:32:49.999583 11805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40081 every 8 connection(s)
I20260928 06:32:49.999920 11738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
I20260928 06:32:50.001206 11806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:32:50.004370 11806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b922dae8c7ab46c89fba21146626e98a: Bootstrap starting.
I20260928 06:32:50.005287 11806 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b922dae8c7ab46c89fba21146626e98a: Neither blocks nor log segments found. Creating new log.
I20260928 06:32:50.005682 11806 log.cc:824] T 00000000000000000000000000000000 P b922dae8c7ab46c89fba21146626e98a: Log is configured to *not* fsync() on all Append() calls
I20260928 06:32:50.006034 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 11738
I20260928 06:32:50.006155 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790576899143097-23430-0/minicluster-data/master-0/wal/instance
I20260928 06:32:50.006357 23430 external_mini_cluster.cc:1428] Setting key c788d3291ea2fd1a336fbaec78622993
I20260928 06:32:50.006583 11806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b922dae8c7ab46c89fba21146626e98a: No bootstrap required, opened a new log
I20260928 06:32:50.008344 11806 raft_consensus.cc:348] T 00000000000000000000000000000000 P b922dae8c7ab46c89fba21146626e98a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b922dae8c7ab46c89fba21146626e98a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40081 } }
I20260928 06:32:50.008507 11806 raft_consensus.cc:374] T 00000000000000000000000000000000 P b922dae8c7ab46c89fba21146626e98a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:32:50.008548 11806 raft_consensus.cc:729] T 00000000000000000000000000000000 P b922dae8c7ab46c89fba21146626e98a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b922dae8c7ab46c89fba21146626e98a, State: Initialized, Role: FOLLOWER
I20260928 06:32:50.008702 11806 consensus_queue.cc:260] T 00000000000000000000000000000000 P b922dae8c7ab46c89fba21146626e98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b922dae8c7ab46c89fba21146626e98a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40081 } }
I20260928 06:32:50.008788 11806 raft_consensus.cc:388] T 00000000000000000000000000000000 P b922dae8c7ab46c89fba21146626e98a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:32:50.008826 11806 raft_consensus.cc:482] T 00000000000000000000000000000000 P b922dae8c7ab46c89fba21146626e98a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:32:50.008872 11806 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b922dae8c7ab46c89fba21146626e98a [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:32:50.009941 11806 raft_consensus.cc:504] T 00000000000000000000000000000000 P b922dae8c7ab46c89fba21146626e98a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b922dae8c7ab46c89fba21146626e98a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40081 } }
I20260928 06:32:50.010063 11806 leader_election.cc:302] T 00000000000000000000000000000000 P b922dae8c7ab46c89fba21146626e98a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b922dae8c7ab46c89fba21146626e98a; no voters:
I20260928 06:32:50.010110 11810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:50.007017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48137 (local address 127.0.0.1:40081)
0928 06:32:50.007179 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:50.007188 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:50.007227 (+ 39us) server_negotiation.cc:407] Connection header received
0928 06:32:50.008325 (+ 1098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:50.008341 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:50.008382 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:50.008537 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:32:50.009473 (+ 936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:32:50.009477 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:32:50.009477 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:32:50.009482 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:32:50.009528 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:32:50.009542 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:32:50.009826 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:50.009863 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 06:32:50.009910 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":77,"threads_started":1}
I20260928 06:32:50.010245 11806 leader_election.cc:288] T 00000000000000000000000000000000 P b922dae8c7ab46c89fba21146626e98a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:32:50.010303 11811 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b922dae8c7ab46c89fba21146626e98a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:32:50.010437 11811 raft_consensus.cc:686] T 00000000000000000000000000000000 P b922dae8c7ab46c89fba21146626e98a [term 1 LEADER]: Becoming Leader. State: Replica: b922dae8c7ab46c89fba21146626e98a, State: Running, Role: LEADER
I20260928 06:32:50.010504 11806 sys_catalog.cc:564] T 00000000000000000000000000000000 P b922dae8c7ab46c89fba21146626e98a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:32:50.010525 11811 consensus_queue.cc:237] T 00000000000000000000000000000000 P b922dae8c7ab46c89fba21146626e98a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b922dae8c7ab46c89fba21146626e98a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40081 } }
I20260928 06:32:50.010864 11811 sys_catalog.cc:455] T 00000000000000000000000000000000 P b922dae8c7ab46c89fba21146626e98a [sys.catalog]: SysCatalogTable state changed. Reason: New leader b922dae8c7ab46c89fba21146626e98a. Latest consensus state: current_term: 1 leader_uuid: "b922dae8c7ab46c89fba21146626e98a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b922dae8c7ab46c89fba21146626e98a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40081 } } }
I20260928 06:32:50.010880 11812 sys_catalog.cc:455] T 00000000000000000000000000000000 P b922dae8c7ab46c89fba21146626e98a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b922dae8c7ab46c89fba21146626e98a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b922dae8c7ab46c89fba21146626e98a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40081 } } }
I20260928 06:32:50.010957 11811 sys_catalog.cc:458] T 00000000000000000000000000000000 P b922dae8c7ab46c89fba21146626e98a [sys.catalog]: This master's current role is: LEADER
I20260928 06:32:50.010990 11812 sys_catalog.cc:458] T 00000000000000000000000000000000 P b922dae8c7ab46c89fba21146626e98a [sys.catalog]: This master's current role is: LEADER
I20260928 06:32:50.011055 23430 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260928 06:32:50.012462 11827 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b922dae8c7ab46c89fba21146626e98a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:32:50.012521 11827 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:32:50.012567 11818 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:32:50.012738 11818 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:32:50.014418 11818 catalog_manager.cc:1347] Generated new cluster ID: e029c56a00f04464af5236e92e154bf5
I20260928 06:32:50.014477 11818 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:32:50.021481 11810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:50.013408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58077 (local address 127.0.0.1:40081)
0928 06:32:50.013443 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:50.013445 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:50.013733 (+ 288us) server_negotiation.cc:407] Connection header received
0928 06:32:50.013784 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:50.013785 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:50.013790 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:50.013853 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:32:50.021197 (+ 7344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:32:50.021198 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:32:50.021199 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:32:50.021205 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:32:50.021222 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:32:50.021227 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:32:50.021259 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:50.021389 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 06:32:50.021406 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:32:50.042802 11818 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:32:50.043040 11818 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:32:50.046213 11818 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b922dae8c7ab46c89fba21146626e98a: Generated new TSK 0
I20260928 06:32:50.046425 11818 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:32:50.055352 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 11738
2026-09-28T06:32:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1466 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790576899143097-23430-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:32:50 dist-test-slave-d59m krb5kdc[11837](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:32:50 dist-test-slave-d59m krb5kdc[11837](info): set up 2 sockets
Sep 28 06:32:50 dist-test-slave-d59m krb5kdc[11837](info): commencing operation
krb5kdc: starting...
W20260928 06:32:51.169328 23430 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Sep 28 06:32:51 dist-test-slave-d59m krb5kdc[11837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:32:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:32:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790576899143097-23430-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790576899143097-23430-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:32:51.296140 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790576899143097-23430-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790576899143097-23430-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790576899143097-23430-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36651
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790576899143097-23430-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36651
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:32:51.384658 11850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:32:51.384891 11850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:32:51.384928 11850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:32:51.386094 11850 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:32:51.386157 11850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:32:51.386183 11850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:32:51.386210 11850 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:32:51.386240 11850 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:32:51.387710 11850 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790576899143097-23430-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790576899143097-23430-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790576899143097-23430-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790576899143097-23430-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:32:51.387986 11850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:32:51.388214 11850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:32:51.393229 11856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:51.393215 11855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:51.393188 11858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:32:51.393445 11850 server_base.cc:1034] running on GCE node
I20260928 06:32:51.393733 11850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:32:51.394034 11850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:32:51.395224 11850 hybrid_clock.cc:648] HybridClock initialized: now 1790577171395208 us; error 34 us; skew 500 ppm
Sep 28 06:32:51 dist-test-slave-d59m krb5kdc[11837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:32:51.398028 11850 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:32:51.398492 11850 webserver.cc:460] Webserver started at http://127.0.0.1:43473/ using document root <none> and password file <none>
I20260928 06:32:51.398753 11850 fs_manager.cc:359] Metadata directory not provided
I20260928 06:32:51.398835 11850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:32:51.398986 11850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:32:51.400054 11850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790576899143097-23430-0/minicluster-data/master-0/data/instance:
uuid: "89cb5dc0581248b0a4b2d9ebb7024fe7"
format_stamp: "Formatted at 2026-09-28 06:32:51 on dist-test-slave-d59m"
server_key: "9aa49b99fc8b7508383e0c6bb25adbbb"
server_key_iv: "6eab0e44ce83282a37e5fc4e1afcefa0"
server_key_version: "encryptionkey@0"
I20260928 06:32:51.400416 11850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790576899143097-23430-0/minicluster-data/master-0/wal/instance:
uuid: "89cb5dc0581248b0a4b2d9ebb7024fe7"
format_stamp: "Formatted at 2026-09-28 06:32:51 on dist-test-slave-d59m"
server_key: "9aa49b99fc8b7508383e0c6bb25adbbb"
server_key_iv: "6eab0e44ce83282a37e5fc4e1afcefa0"
server_key_version: "encryptionkey@0"
I20260928 06:32:51.401912 11850 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:32:51.402714 11865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:51.402910 11850 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:32:51.403024 11850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790576899143097-23430-0/minicluster-data/master-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790576899143097-23430-0/minicluster-data/master-0/wal
uuid: "89cb5dc0581248b0a4b2d9ebb7024fe7"
format_stamp: "Formatted at 2026-09-28 06:32:51 on dist-test-slave-d59m"
server_key: "9aa49b99fc8b7508383e0c6bb25adbbb"
server_key_iv: "6eab0e44ce83282a37e5fc4e1afcefa0"
server_key_version: "encryptionkey@0"
I20260928 06:32:51.403101 11850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790576899143097-23430-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790576899143097-23430-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790576899143097-23430-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:32:51.428565 11850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:32:51.431028 11850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:32:51.431188 11850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:32:51.435491 11850 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36651
I20260928 06:32:51.435542 11917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36651 every 8 connection(s)
I20260928 06:32:51.435921 11850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
I20260928 06:32:51.436957 11918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:32:51.440091 11918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89cb5dc0581248b0a4b2d9ebb7024fe7: Bootstrap starting.
I20260928 06:32:51.440806 11918 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89cb5dc0581248b0a4b2d9ebb7024fe7: Neither blocks nor log segments found. Creating new log.
I20260928 06:32:51.441145 11918 log.cc:824] T 00000000000000000000000000000000 P 89cb5dc0581248b0a4b2d9ebb7024fe7: Log is configured to *not* fsync() on all Append() calls
I20260928 06:32:51.442036 11918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89cb5dc0581248b0a4b2d9ebb7024fe7: No bootstrap required, opened a new log
I20260928 06:32:51.442152 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 11850
I20260928 06:32:51.442282 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790576899143097-23430-0/minicluster-data/master-0/wal/instance
I20260928 06:32:51.442474 23430 external_mini_cluster.cc:1428] Setting key b08eb1b3d6a15f22121426419870f191
I20260928 06:32:51.444499 11918 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89cb5dc0581248b0a4b2d9ebb7024fe7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89cb5dc0581248b0a4b2d9ebb7024fe7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36651 } }
I20260928 06:32:51.444706 11918 raft_consensus.cc:374] T 00000000000000000000000000000000 P 89cb5dc0581248b0a4b2d9ebb7024fe7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:32:51.444747 11918 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89cb5dc0581248b0a4b2d9ebb7024fe7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89cb5dc0581248b0a4b2d9ebb7024fe7, State: Initialized, Role: FOLLOWER
I20260928 06:32:51.444883 11918 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89cb5dc0581248b0a4b2d9ebb7024fe7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89cb5dc0581248b0a4b2d9ebb7024fe7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36651 } }
I20260928 06:32:51.444964 11918 raft_consensus.cc:388] T 00000000000000000000000000000000 P 89cb5dc0581248b0a4b2d9ebb7024fe7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:32:51.445019 11918 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89cb5dc0581248b0a4b2d9ebb7024fe7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:32:51.445065 11918 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89cb5dc0581248b0a4b2d9ebb7024fe7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:32:51.446219 11918 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89cb5dc0581248b0a4b2d9ebb7024fe7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89cb5dc0581248b0a4b2d9ebb7024fe7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36651 } }
I20260928 06:32:51.446368 11918 leader_election.cc:302] T 00000000000000000000000000000000 P 89cb5dc0581248b0a4b2d9ebb7024fe7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89cb5dc0581248b0a4b2d9ebb7024fe7; no voters:
I20260928 06:32:51.446542 11918 leader_election.cc:288] T 00000000000000000000000000000000 P 89cb5dc0581248b0a4b2d9ebb7024fe7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:32:51.446646 11923 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89cb5dc0581248b0a4b2d9ebb7024fe7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:32:51.446813 11918 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89cb5dc0581248b0a4b2d9ebb7024fe7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:32:51.446830 11923 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89cb5dc0581248b0a4b2d9ebb7024fe7 [term 1 LEADER]: Becoming Leader. State: Replica: 89cb5dc0581248b0a4b2d9ebb7024fe7, State: Running, Role: LEADER
I20260928 06:32:51.447014 11923 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89cb5dc0581248b0a4b2d9ebb7024fe7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89cb5dc0581248b0a4b2d9ebb7024fe7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36651 } }
I20260928 06:32:51.447404 11923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89cb5dc0581248b0a4b2d9ebb7024fe7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "89cb5dc0581248b0a4b2d9ebb7024fe7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89cb5dc0581248b0a4b2d9ebb7024fe7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36651 } } }
I20260928 06:32:51.447532 11923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89cb5dc0581248b0a4b2d9ebb7024fe7 [sys.catalog]: This master's current role is: LEADER
I20260928 06:32:51.447715 11923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89cb5dc0581248b0a4b2d9ebb7024fe7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89cb5dc0581248b0a4b2d9ebb7024fe7. Latest consensus state: current_term: 1 leader_uuid: "89cb5dc0581248b0a4b2d9ebb7024fe7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89cb5dc0581248b0a4b2d9ebb7024fe7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36651 } } }
I20260928 06:32:51.447818 11923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89cb5dc0581248b0a4b2d9ebb7024fe7 [sys.catalog]: This master's current role is: LEADER
I20260928 06:32:51.447966 11932 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:32:51.448568 11932 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:32:51.450270 11932 catalog_manager.cc:1347] Generated new cluster ID: 5d91441dd16b492e9ad8a7e017139937
I20260928 06:32:51.450348 11932 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:32:51 dist-test-slave-d59m krb5kdc[11837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 06:32:51.460947 11922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:51.443027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.87:49657 (local address 127.0.0.1:36651)
0928 06:32:51.443195 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:51.443202 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:51.443228 (+ 26us) server_negotiation.cc:407] Connection header received
0928 06:32:51.444460 (+ 1232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:51.444468 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:51.444525 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:51.444877 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:32:51.445508 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:51.446490 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:51.447598 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:51.448284 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:51.448393 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:51.452006 (+ 3613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:32:51.452009 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:32:51.452010 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:32:51.452024 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:32:51.458610 (+ 6586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:32:51.459150 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:32:51.459154 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:32:51.459154 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:32:51.459267 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:32:51.459514 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:32:51.459514 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:32:51.459515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:32:51.459847 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:32:51.460015 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:51.460508 (+ 493us) server_negotiation.cc:299] Negotiation successful
0928 06:32:51.460686 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":72,"threads_started":1}
I20260928 06:32:51.461879 23430 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:32:51.465843 11932 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:32:51.466632 11932 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:32:51.471401 11922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:51.463401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.87:57301 (local address 127.0.0.1:36651)
0928 06:32:51.463435 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:51.463437 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:51.463693 (+ 256us) server_negotiation.cc:407] Connection header received
0928 06:32:51.463802 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:51.463803 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:51.463813 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:51.463842 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:32:51.464529 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:51.465339 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:51.466376 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:51.466699 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:51.466742 (+ 43us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:51.468308 (+ 1566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:32:51.468309 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:32:51.468309 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:32:51.468316 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:32:51.470049 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:32:51.470494 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:32:51.470496 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:32:51.470496 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:32:51.470553 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:32:51.470796 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:32:51.470797 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:32:51.470797 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:32:51.470916 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:32:51.470992 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:51.471237 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 06:32:51.471309 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 06:32:51.474668 11932 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 89cb5dc0581248b0a4b2d9ebb7024fe7: Generated new TSK 0
I20260928 06:32:51.474896 11932 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:32:51.479568 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 11850
W20260928 06:32:51.483961 11835 connection.cc:535] client connection to 127.0.0.1:36651 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36651): Connection reset by peer (error 104)
2026-09-28T06:32:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1422 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790576899143097-23430-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:32:51 dist-test-slave-d59m krb5kdc[11949](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:32:51 dist-test-slave-d59m krb5kdc[11949](info): set up 2 sockets
Sep 28 06:32:51 dist-test-slave-d59m krb5kdc[11949](info): commencing operation
krb5kdc: starting...
W20260928 06:32:52.541129 23430 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Sep 28 06:32:52 dist-test-slave-d59m krb5kdc[11949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:32:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:32:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790576899143097-23430-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790576899143097-23430-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:32:52.661465 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790576899143097-23430-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790576899143097-23430-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790576899143097-23430-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43201
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790576899143097-23430-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43201
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:32:52.764555 11962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:32:52.764834 11962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:32:52.764864 11962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:32:52.766036 11962 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:32:52.766079 11962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:32:52.766094 11962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:32:52.766108 11962 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:32:52.766158 11962 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:32:52.767652 11962 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790576899143097-23430-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790576899143097-23430-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43201
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790576899143097-23430-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790576899143097-23430-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:32:52.767920 11962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:32:52.768144 11962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:32:52.772964 11970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:52.772956 11967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:52.772956 11968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:32:52.773272 11962 server_base.cc:1034] running on GCE node
I20260928 06:32:52.773550 11962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:32:52.773844 11962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:32:52.775009 11962 hybrid_clock.cc:648] HybridClock initialized: now 1790577172774991 us; error 35 us; skew 500 ppm
Sep 28 06:32:52 dist-test-slave-d59m krb5kdc[11949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:32:52.777586 11962 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:32:52.778085 11962 webserver.cc:460] Webserver started at http://127.0.0.1:36725/ using document root <none> and password file <none>
I20260928 06:32:52.778324 11962 fs_manager.cc:359] Metadata directory not provided
I20260928 06:32:52.778402 11962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:32:52.778555 11962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:32:52.779665 11962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790576899143097-23430-0/minicluster-data/master-0/data/instance:
uuid: "e79329a48013462d92ae69b547919c65"
format_stamp: "Formatted at 2026-09-28 06:32:52 on dist-test-slave-d59m"
server_key: "3404158e59cb91c64deb1e10ce787706"
server_key_iv: "f3056b1af7b102284b45187f57f9f706"
server_key_version: "encryptionkey@0"
I20260928 06:32:52.780043 11962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790576899143097-23430-0/minicluster-data/master-0/wal/instance:
uuid: "e79329a48013462d92ae69b547919c65"
format_stamp: "Formatted at 2026-09-28 06:32:52 on dist-test-slave-d59m"
server_key: "3404158e59cb91c64deb1e10ce787706"
server_key_iv: "f3056b1af7b102284b45187f57f9f706"
server_key_version: "encryptionkey@0"
I20260928 06:32:52.781566 11962 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:32:52.782371 11977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:52.782532 11962 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:32:52.782636 11962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790576899143097-23430-0/minicluster-data/master-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790576899143097-23430-0/minicluster-data/master-0/wal
uuid: "e79329a48013462d92ae69b547919c65"
format_stamp: "Formatted at 2026-09-28 06:32:52 on dist-test-slave-d59m"
server_key: "3404158e59cb91c64deb1e10ce787706"
server_key_iv: "f3056b1af7b102284b45187f57f9f706"
server_key_version: "encryptionkey@0"
I20260928 06:32:52.782709 11962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790576899143097-23430-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790576899143097-23430-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790576899143097-23430-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:32:52.803742 11962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:32:52.806265 11962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:32:52.806440 11962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:32:52.811192 11962 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43201
I20260928 06:32:52.811220 12029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43201 every 8 connection(s)
I20260928 06:32:52.811645 11962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
I20260928 06:32:52.812526 12030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:32:52.814829 12030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e79329a48013462d92ae69b547919c65: Bootstrap starting.
I20260928 06:32:52.815433 12030 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e79329a48013462d92ae69b547919c65: Neither blocks nor log segments found. Creating new log.
I20260928 06:32:52.815712 12030 log.cc:824] T 00000000000000000000000000000000 P e79329a48013462d92ae69b547919c65: Log is configured to *not* fsync() on all Append() calls
I20260928 06:32:52.816466 12030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e79329a48013462d92ae69b547919c65: No bootstrap required, opened a new log
I20260928 06:32:52.816867 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 11962
I20260928 06:32:52.816982 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790576899143097-23430-0/minicluster-data/master-0/wal/instance
I20260928 06:32:52.817178 23430 external_mini_cluster.cc:1428] Setting key 1e2e3fa473e1bbec67c1343ae4525d2c
I20260928 06:32:52.818783 12030 raft_consensus.cc:348] T 00000000000000000000000000000000 P e79329a48013462d92ae69b547919c65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e79329a48013462d92ae69b547919c65" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43201 } }
I20260928 06:32:52.819000 12030 raft_consensus.cc:374] T 00000000000000000000000000000000 P e79329a48013462d92ae69b547919c65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:32:52.819036 12030 raft_consensus.cc:729] T 00000000000000000000000000000000 P e79329a48013462d92ae69b547919c65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e79329a48013462d92ae69b547919c65, State: Initialized, Role: FOLLOWER
I20260928 06:32:52.819233 12030 consensus_queue.cc:260] T 00000000000000000000000000000000 P e79329a48013462d92ae69b547919c65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e79329a48013462d92ae69b547919c65" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43201 } }
I20260928 06:32:52.819360 12030 raft_consensus.cc:388] T 00000000000000000000000000000000 P e79329a48013462d92ae69b547919c65 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:32:52.819428 12030 raft_consensus.cc:482] T 00000000000000000000000000000000 P e79329a48013462d92ae69b547919c65 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:32:52.819481 12030 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e79329a48013462d92ae69b547919c65 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:32:52.820015 12030 raft_consensus.cc:504] T 00000000000000000000000000000000 P e79329a48013462d92ae69b547919c65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e79329a48013462d92ae69b547919c65" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43201 } }
I20260928 06:32:52.820132 12030 leader_election.cc:302] T 00000000000000000000000000000000 P e79329a48013462d92ae69b547919c65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e79329a48013462d92ae69b547919c65; no voters:
I20260928 06:32:52.820289 12030 leader_election.cc:288] T 00000000000000000000000000000000 P e79329a48013462d92ae69b547919c65 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:32:52.820322 12036 raft_consensus.cc:482] T 00000000000000000000000000000000 P e79329a48013462d92ae69b547919c65 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:32:52.820474 12036 raft_consensus.cc:504] T 00000000000000000000000000000000 P e79329a48013462d92ae69b547919c65 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e79329a48013462d92ae69b547919c65" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43201 } }
I20260928 06:32:52.820552 12030 sys_catalog.cc:564] T 00000000000000000000000000000000 P e79329a48013462d92ae69b547919c65 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:32:52.821990 12036 leader_election.cc:302] T 00000000000000000000000000000000 P e79329a48013462d92ae69b547919c65 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e79329a48013462d92ae69b547919c65; no voters:
I20260928 06:32:52.822007 12035 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e79329a48013462d92ae69b547919c65 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:32:52.822167 12036 leader_election.cc:288] T 00000000000000000000000000000000 P e79329a48013462d92ae69b547919c65 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers
I20260928 06:32:52.822180 12035 raft_consensus.cc:686] T 00000000000000000000000000000000 P e79329a48013462d92ae69b547919c65 [term 1 LEADER]: Becoming Leader. State: Replica: e79329a48013462d92ae69b547919c65, State: Running, Role: LEADER
I20260928 06:32:52.822300 12035 consensus_queue.cc:237] T 00000000000000000000000000000000 P e79329a48013462d92ae69b547919c65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e79329a48013462d92ae69b547919c65" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43201 } }
I20260928 06:32:52.822758 12036 sys_catalog.cc:455] T 00000000000000000000000000000000 P e79329a48013462d92ae69b547919c65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e79329a48013462d92ae69b547919c65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e79329a48013462d92ae69b547919c65" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43201 } } }
I20260928 06:32:52.822953 12036 sys_catalog.cc:458] T 00000000000000000000000000000000 P e79329a48013462d92ae69b547919c65 [sys.catalog]: This master's current role is: LEADER
I20260928 06:32:52.822780 12035 sys_catalog.cc:455] T 00000000000000000000000000000000 P e79329a48013462d92ae69b547919c65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e79329a48013462d92ae69b547919c65. Latest consensus state: current_term: 1 leader_uuid: "e79329a48013462d92ae69b547919c65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e79329a48013462d92ae69b547919c65" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43201 } } }
I20260928 06:32:52.823056 12035 sys_catalog.cc:458] T 00000000000000000000000000000000 P e79329a48013462d92ae69b547919c65 [sys.catalog]: This master's current role is: LEADER
W20260928 06:32:52.823738 12051 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e79329a48013462d92ae69b547919c65: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:32:52.823840 12051 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:32:52.824011 12052 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:32:52.824229 12052 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:32:52.824250 12034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:52.817691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.87:52671 (local address 127.0.0.1:43201)
0928 06:32:52.817889 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:52.817898 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:52.817995 (+ 97us) server_negotiation.cc:407] Connection header received
0928 06:32:52.819000 (+ 1005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:52.819009 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:52.819079 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:52.819239 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:32:52.819514 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:52.820815 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:52.821814 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:52.822387 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:52.822488 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:52.823511 (+ 1023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:32:52.823515 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:32:52.823515 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:32:52.823719 (+ 204us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:32:52.823757 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:32:52.823772 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:32:52.823842 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:52.824039 (+ 197us) server_negotiation.cc:299] Negotiation successful
0928 06:32:52.824065 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":80,"threads_started":1}
I20260928 06:32:52.825214 23430 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:32:52.826071 12052 catalog_manager.cc:1347] Generated new cluster ID: e6e76acfa8c14702973d3d0f0b152efc
I20260928 06:32:52.826112 12052 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:32:52.831986 12034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:52.826712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.87:39533 (local address 127.0.0.1:43201)
0928 06:32:52.826756 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:52.826758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:52.828722 (+ 1964us) server_negotiation.cc:407] Connection header received
0928 06:32:52.828805 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:52.828806 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:52.828815 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:52.828841 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:32:52.829025 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:52.829718 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:52.830682 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:52.830972 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:52.831006 (+ 34us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:52.831728 (+ 722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:32:52.831729 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:32:52.831729 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:32:52.831733 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:32:52.831746 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:32:52.831750 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:32:52.831778 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:52.831902 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 06:32:52.831913 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:32:52.841281 12052 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:32:52.842001 12052 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:32:52.848016 12052 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e79329a48013462d92ae69b547919c65: Generated new TSK 0
I20260928 06:32:52.848239 12052 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:32:52.853258 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 11962
W20260928 06:32:52.858819 11947 connection.cc:535] client connection to 127.0.0.1:43201 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43201): Connection reset by peer (error 104)
2026-09-28T06:32:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1374 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790576899143097-23430-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:32:52 dist-test-slave-d59m krb5kdc[12063](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:32:52 dist-test-slave-d59m krb5kdc[12063](info): set up 2 sockets
Sep 28 06:32:52 dist-test-slave-d59m krb5kdc[12063](info): commencing operation
krb5kdc: starting...
W20260928 06:32:53.917249 23430 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Sep 28 06:32:53 dist-test-slave-d59m krb5kdc[12063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:32:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:32:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790576899143097-23430-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790576899143097-23430-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:32:54.041582 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790576899143097-23430-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790576899143097-23430-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790576899143097-23430-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38607
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790576899143097-23430-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38607
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:32:54.125031 12076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:32:54.125272 12076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:32:54.125298 12076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:32:54.126577 12076 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:32:54.126628 12076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:32:54.126642 12076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:32:54.126655 12076 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:32:54.126667 12076 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:32:54.128208 12076 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790576899143097-23430-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790576899143097-23430-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38607
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790576899143097-23430-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12076
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790576899143097-23430-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:32:54.128481 12076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:32:54.128746 12076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:32:54.133673 12084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:32:54.133898 12076 server_base.cc:1034] running on GCE node
W20260928 06:32:54.133695 12081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:54.133680 12082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:32:54.134282 12076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:32:54.134590 12076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:32:54.135754 12076 hybrid_clock.cc:648] HybridClock initialized: now 1790577174135734 us; error 38 us; skew 500 ppm
Sep 28 06:32:54 dist-test-slave-d59m krb5kdc[12063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577174, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:32:54.138586 12076 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:32:54.139178 12076 webserver.cc:460] Webserver started at http://127.0.0.1:45791/ using document root <none> and password file <none>
I20260928 06:32:54.139442 12076 fs_manager.cc:359] Metadata directory not provided
I20260928 06:32:54.139510 12076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:32:54.139658 12076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:32:54.141179 12076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790576899143097-23430-0/minicluster-data/master-0/data/instance:
uuid: "f59e89f655f04571b35ddc27105ab3ad"
format_stamp: "Formatted at 2026-09-28 06:32:54 on dist-test-slave-d59m"
server_key: "154131eede664d74e899e48708c74497"
server_key_iv: "157c0893c726f2a4719892ee5ddfd0b0"
server_key_version: "encryptionkey@0"
I20260928 06:32:54.141607 12076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790576899143097-23430-0/minicluster-data/master-0/wal/instance:
uuid: "f59e89f655f04571b35ddc27105ab3ad"
format_stamp: "Formatted at 2026-09-28 06:32:54 on dist-test-slave-d59m"
server_key: "154131eede664d74e899e48708c74497"
server_key_iv: "157c0893c726f2a4719892ee5ddfd0b0"
server_key_version: "encryptionkey@0"
I20260928 06:32:54.143326 12076 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:32:54.144204 12091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:54.144429 12076 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:32:54.144517 12076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790576899143097-23430-0/minicluster-data/master-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790576899143097-23430-0/minicluster-data/master-0/wal
uuid: "f59e89f655f04571b35ddc27105ab3ad"
format_stamp: "Formatted at 2026-09-28 06:32:54 on dist-test-slave-d59m"
server_key: "154131eede664d74e899e48708c74497"
server_key_iv: "157c0893c726f2a4719892ee5ddfd0b0"
server_key_version: "encryptionkey@0"
I20260928 06:32:54.144641 12076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790576899143097-23430-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790576899143097-23430-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790576899143097-23430-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:32:54.148025 12076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:32:54.150265 12076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:32:54.150449 12076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:32:54.154313 12076 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38607
I20260928 06:32:54.154367 12143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38607 every 8 connection(s)
I20260928 06:32:54.154729 12076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
I20260928 06:32:54.155930 12144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:32:54.156313 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 12076
I20260928 06:32:54.156451 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790576899143097-23430-0/minicluster-data/master-0/wal/instance
I20260928 06:32:54.156733 23430 external_mini_cluster.cc:1428] Setting key 3f6b1bc4f44c675ec2b3cead22ed6ebd
I20260928 06:32:54.159384 12147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:54.157289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.87:57045 (local address 127.0.0.1:38607)
0928 06:32:54.157511 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:54.157522 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:54.157602 (+ 80us) server_negotiation.cc:407] Connection header received
0928 06:32:54.158077 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:54.158084 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:54.158110 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:54.158279 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:32:54.158936 (+ 657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:32:54.158940 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:32:54.158941 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:32:54.158945 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:32:54.159000 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:32:54.159010 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:32:54.159083 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:54.159150 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 06:32:54.159196 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":80,"threads_started":1}
I20260928 06:32:54.159477 12144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f59e89f655f04571b35ddc27105ab3ad: Bootstrap starting.
I20260928 06:32:54.160259 12144 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f59e89f655f04571b35ddc27105ab3ad: Neither blocks nor log segments found. Creating new log.
I20260928 06:32:54.160334 23430 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:32:54.160632 12144 log.cc:824] T 00000000000000000000000000000000 P f59e89f655f04571b35ddc27105ab3ad: Log is configured to *not* fsync() on all Append() calls
I20260928 06:32:54.161475 12144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f59e89f655f04571b35ddc27105ab3ad: No bootstrap required, opened a new log
I20260928 06:32:54.162956 12147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:54.161887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.87:57033 (local address 127.0.0.1:38607)
0928 06:32:54.161923 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:54.161925 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:54.162041 (+ 116us) server_negotiation.cc:407] Connection header received
0928 06:32:54.162110 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:54.162111 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:54.162114 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:54.162183 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:32:54.162720 (+ 537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:32:54.162721 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:32:54.162721 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:32:54.162726 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:32:54.162738 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:32:54.162742 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:32:54.162799 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:54.162854 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 06:32:54.162869 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:32:54.164324 12144 raft_consensus.cc:348] T 00000000000000000000000000000000 P f59e89f655f04571b35ddc27105ab3ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f59e89f655f04571b35ddc27105ab3ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38607 } }
I20260928 06:32:54.164508 12144 raft_consensus.cc:374] T 00000000000000000000000000000000 P f59e89f655f04571b35ddc27105ab3ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:32:54.164620 12144 raft_consensus.cc:729] T 00000000000000000000000000000000 P f59e89f655f04571b35ddc27105ab3ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f59e89f655f04571b35ddc27105ab3ad, State: Initialized, Role: FOLLOWER
I20260928 06:32:54.164762 12144 consensus_queue.cc:260] T 00000000000000000000000000000000 P f59e89f655f04571b35ddc27105ab3ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f59e89f655f04571b35ddc27105ab3ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38607 } }
I20260928 06:32:54.164872 12144 raft_consensus.cc:388] T 00000000000000000000000000000000 P f59e89f655f04571b35ddc27105ab3ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:32:54.164922 12144 raft_consensus.cc:482] T 00000000000000000000000000000000 P f59e89f655f04571b35ddc27105ab3ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:32:54.164973 12144 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f59e89f655f04571b35ddc27105ab3ad [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:32:54.165683 12144 raft_consensus.cc:504] T 00000000000000000000000000000000 P f59e89f655f04571b35ddc27105ab3ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f59e89f655f04571b35ddc27105ab3ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38607 } }
I20260928 06:32:54.165791 12144 leader_election.cc:302] T 00000000000000000000000000000000 P f59e89f655f04571b35ddc27105ab3ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f59e89f655f04571b35ddc27105ab3ad; no voters:
I20260928 06:32:54.165977 12144 leader_election.cc:288] T 00000000000000000000000000000000 P f59e89f655f04571b35ddc27105ab3ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:32:54.166040 12154 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f59e89f655f04571b35ddc27105ab3ad [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:32:54.166209 12154 raft_consensus.cc:686] T 00000000000000000000000000000000 P f59e89f655f04571b35ddc27105ab3ad [term 1 LEADER]: Becoming Leader. State: Replica: f59e89f655f04571b35ddc27105ab3ad, State: Running, Role: LEADER
I20260928 06:32:54.166222 12144 sys_catalog.cc:564] T 00000000000000000000000000000000 P f59e89f655f04571b35ddc27105ab3ad [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:32:54.166350 12154 consensus_queue.cc:237] T 00000000000000000000000000000000 P f59e89f655f04571b35ddc27105ab3ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f59e89f655f04571b35ddc27105ab3ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38607 } }
I20260928 06:32:54.166723 12154 sys_catalog.cc:455] T 00000000000000000000000000000000 P f59e89f655f04571b35ddc27105ab3ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader f59e89f655f04571b35ddc27105ab3ad. Latest consensus state: current_term: 1 leader_uuid: "f59e89f655f04571b35ddc27105ab3ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f59e89f655f04571b35ddc27105ab3ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38607 } } }
I20260928 06:32:54.166800 12154 sys_catalog.cc:458] T 00000000000000000000000000000000 P f59e89f655f04571b35ddc27105ab3ad [sys.catalog]: This master's current role is: LEADER
I20260928 06:32:54.166939 12156 sys_catalog.cc:455] T 00000000000000000000000000000000 P f59e89f655f04571b35ddc27105ab3ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f59e89f655f04571b35ddc27105ab3ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f59e89f655f04571b35ddc27105ab3ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38607 } } }
I20260928 06:32:54.167013 12156 sys_catalog.cc:458] T 00000000000000000000000000000000 P f59e89f655f04571b35ddc27105ab3ad [sys.catalog]: This master's current role is: LEADER
W20260928 06:32:54.168103 12169 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f59e89f655f04571b35ddc27105ab3ad: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:32:54.168165 12169 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:32:54.168553 12161 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:32:54.168768 12161 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:32:54.175540 12161 catalog_manager.cc:1347] Generated new cluster ID: 4d4899b986ef4918ba4daa7143392171
I20260928 06:32:54.175603 12161 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:32:54.199298 12161 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:32:54.199517 12161 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:32:54.203218 12161 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f59e89f655f04571b35ddc27105ab3ad: Generated new TSK 0
I20260928 06:32:54.203426 12161 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:32:54.206226 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 12076
2026-09-28T06:32:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1353 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790576899143097-23430-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:32:54 dist-test-slave-d59m krb5kdc[12175](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:32:54 dist-test-slave-d59m krb5kdc[12175](info): set up 2 sockets
Sep 28 06:32:54 dist-test-slave-d59m krb5kdc[12175](info): commencing operation
krb5kdc: starting...
W20260928 06:32:55.305352 23430 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Sep 28 06:32:55 dist-test-slave-d59m krb5kdc[12175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:32:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:32:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790576899143097-23430-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790576899143097-23430-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:32:55.451309 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790576899143097-23430-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790576899143097-23430-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790576899143097-23430-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35515
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790576899143097-23430-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35515
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:32:55.548525 12188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:32:55.552686 12188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:32:55.552742 12188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:32:55.555296 12188 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:32:55.555371 12188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:32:55.555413 12188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:32:55.555436 12188 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:32:55.555469 12188 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:32:55.558477 12188 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790576899143097-23430-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790576899143097-23430-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35515
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790576899143097-23430-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790576899143097-23430-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:32:55.558782 12188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:32:55.559072 12188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:32:55.565407 12194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:55.565416 12193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:55.565596 12196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:32:55.565629 12188 server_base.cc:1034] running on GCE node
I20260928 06:32:55.566005 12188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:32:55.566375 12188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:32:55.567581 12188 hybrid_clock.cc:648] HybridClock initialized: now 1790577175567552 us; error 44 us; skew 500 ppm
Sep 28 06:32:55 dist-test-slave-d59m krb5kdc[12175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577175, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:32:55.576539 12188 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:32:55.577071 12188 webserver.cc:460] Webserver started at http://127.0.0.1:41307/ using document root <none> and password file <none>
I20260928 06:32:55.577322 12188 fs_manager.cc:359] Metadata directory not provided
I20260928 06:32:55.577389 12188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:32:55.577562 12188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:32:55.578737 12188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790576899143097-23430-0/minicluster-data/master-0/data/instance:
uuid: "9ff39fa7ce5f4c61ab3ad8ed59ad4a63"
format_stamp: "Formatted at 2026-09-28 06:32:55 on dist-test-slave-d59m"
server_key: "f42a0091a2e2211873afff945c2eb6fd"
server_key_iv: "6368f0860c7c6d2098adc836f9fff41b"
server_key_version: "encryptionkey@0"
I20260928 06:32:55.579115 12188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790576899143097-23430-0/minicluster-data/master-0/wal/instance:
uuid: "9ff39fa7ce5f4c61ab3ad8ed59ad4a63"
format_stamp: "Formatted at 2026-09-28 06:32:55 on dist-test-slave-d59m"
server_key: "f42a0091a2e2211873afff945c2eb6fd"
server_key_iv: "6368f0860c7c6d2098adc836f9fff41b"
server_key_version: "encryptionkey@0"
I20260928 06:32:55.580530 12188 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 06:32:55.581362 12203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:55.581552 12188 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:32:55.581653 12188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790576899143097-23430-0/minicluster-data/master-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790576899143097-23430-0/minicluster-data/master-0/wal
uuid: "9ff39fa7ce5f4c61ab3ad8ed59ad4a63"
format_stamp: "Formatted at 2026-09-28 06:32:55 on dist-test-slave-d59m"
server_key: "f42a0091a2e2211873afff945c2eb6fd"
server_key_iv: "6368f0860c7c6d2098adc836f9fff41b"
server_key_version: "encryptionkey@0"
I20260928 06:32:55.581737 12188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790576899143097-23430-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790576899143097-23430-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790576899143097-23430-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:32:55.590061 12188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:32:55.592274 12188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:32:55.592419 12188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:32:55.596796 12188 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35515
I20260928 06:32:55.596876 12255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35515 every 8 connection(s)
I20260928 06:32:55.597203 12188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
I20260928 06:32:55.598313 12256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:32:55.601780 12256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ff39fa7ce5f4c61ab3ad8ed59ad4a63: Bootstrap starting.
I20260928 06:32:55.602509 12256 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ff39fa7ce5f4c61ab3ad8ed59ad4a63: Neither blocks nor log segments found. Creating new log.
I20260928 06:32:55.602849 12256 log.cc:824] T 00000000000000000000000000000000 P 9ff39fa7ce5f4c61ab3ad8ed59ad4a63: Log is configured to *not* fsync() on all Append() calls
I20260928 06:32:55.603649 12256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ff39fa7ce5f4c61ab3ad8ed59ad4a63: No bootstrap required, opened a new log
I20260928 06:32:55.606220 12256 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ff39fa7ce5f4c61ab3ad8ed59ad4a63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ff39fa7ce5f4c61ab3ad8ed59ad4a63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35515 } }
I20260928 06:32:55.606408 12256 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ff39fa7ce5f4c61ab3ad8ed59ad4a63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:32:55.606451 12256 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ff39fa7ce5f4c61ab3ad8ed59ad4a63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ff39fa7ce5f4c61ab3ad8ed59ad4a63, State: Initialized, Role: FOLLOWER
I20260928 06:32:55.606609 12256 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ff39fa7ce5f4c61ab3ad8ed59ad4a63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ff39fa7ce5f4c61ab3ad8ed59ad4a63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35515 } }
I20260928 06:32:55.606694 12256 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ff39fa7ce5f4c61ab3ad8ed59ad4a63 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:32:55.606743 12256 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ff39fa7ce5f4c61ab3ad8ed59ad4a63 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:32:55.606796 12256 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ff39fa7ce5f4c61ab3ad8ed59ad4a63 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:32:55.607308 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 12188
I20260928 06:32:55.607421 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790576899143097-23430-0/minicluster-data/master-0/wal/instance
I20260928 06:32:55.607618 23430 external_mini_cluster.cc:1428] Setting key de002abb88c80b325985d5be76049cd7
I20260928 06:32:55.607587 12256 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ff39fa7ce5f4c61ab3ad8ed59ad4a63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ff39fa7ce5f4c61ab3ad8ed59ad4a63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35515 } }
I20260928 06:32:55.607700 12256 leader_election.cc:302] T 00000000000000000000000000000000 P 9ff39fa7ce5f4c61ab3ad8ed59ad4a63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ff39fa7ce5f4c61ab3ad8ed59ad4a63; no voters:
I20260928 06:32:55.607880 12256 leader_election.cc:288] T 00000000000000000000000000000000 P 9ff39fa7ce5f4c61ab3ad8ed59ad4a63 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:32:55.607928 12259 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ff39fa7ce5f4c61ab3ad8ed59ad4a63 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:32:55.608093 12259 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ff39fa7ce5f4c61ab3ad8ed59ad4a63 [term 1 LEADER]: Becoming Leader. State: Replica: 9ff39fa7ce5f4c61ab3ad8ed59ad4a63, State: Running, Role: LEADER
I20260928 06:32:55.608145 12256 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ff39fa7ce5f4c61ab3ad8ed59ad4a63 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:32:55.608182 12259 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ff39fa7ce5f4c61ab3ad8ed59ad4a63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ff39fa7ce5f4c61ab3ad8ed59ad4a63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35515 } }
I20260928 06:32:55.608567 12259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ff39fa7ce5f4c61ab3ad8ed59ad4a63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ff39fa7ce5f4c61ab3ad8ed59ad4a63. Latest consensus state: current_term: 1 leader_uuid: "9ff39fa7ce5f4c61ab3ad8ed59ad4a63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ff39fa7ce5f4c61ab3ad8ed59ad4a63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35515 } } }
I20260928 06:32:55.608883 12259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ff39fa7ce5f4c61ab3ad8ed59ad4a63 [sys.catalog]: This master's current role is: LEADER
I20260928 06:32:55.608781 12260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ff39fa7ce5f4c61ab3ad8ed59ad4a63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ff39fa7ce5f4c61ab3ad8ed59ad4a63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ff39fa7ce5f4c61ab3ad8ed59ad4a63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35515 } } }
I20260928 06:32:55.609249 12260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ff39fa7ce5f4c61ab3ad8ed59ad4a63 [sys.catalog]: This master's current role is: LEADER
W20260928 06:32:55.610169 12277 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9ff39fa7ce5f4c61ab3ad8ed59ad4a63: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:32:55.610226 12277 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:32:55.610289 12270 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:32:55.610505 12270 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:32:55.612102 12270 catalog_manager.cc:1347] Generated new cluster ID: 84d5c2fc275a4837a012025cb74f05a9
I20260928 06:32:55.612154 12270 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:32:55 dist-test-slave-d59m krb5kdc[12175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 06:32:55.624020 12263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:55.608263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.87:41213 (local address 127.0.0.1:35515)
0928 06:32:55.608803 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:55.608814 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:55.608849 (+ 35us) server_negotiation.cc:407] Connection header received
0928 06:32:55.609083 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:55.609092 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:55.609157 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:55.609383 (+ 226us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:32:55.610517 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:55.611666 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:55.612678 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:55.613106 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:55.613225 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:55.615579 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:32:55.615583 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:32:55.615584 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:32:55.615595 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:32:55.621708 (+ 6113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:32:55.622243 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:32:55.622247 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:32:55.622247 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:32:55.622370 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:32:55.622626 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:32:55.622627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:32:55.622627 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:32:55.622947 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:32:55.623135 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:55.623664 (+ 529us) server_negotiation.cc:299] Negotiation successful
0928 06:32:55.623775 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":71,"threads_started":1}
I20260928 06:32:55.625023 23430 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:32:55.634806 12263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:55.626538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.87:40223 (local address 127.0.0.1:35515)
0928 06:32:55.626572 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:55.626573 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:55.626698 (+ 125us) server_negotiation.cc:407] Connection header received
0928 06:32:55.626808 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:55.626809 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:55.626819 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:55.626847 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:32:55.627553 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:55.628319 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:55.629511 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:55.629895 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:55.629938 (+ 43us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:55.631770 (+ 1832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:32:55.631772 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:32:55.631772 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:32:55.631779 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:32:55.633485 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:32:55.633916 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:32:55.633917 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:32:55.633917 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:32:55.633979 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:32:55.634211 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:32:55.634212 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:32:55.634212 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:32:55.634333 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:32:55.634405 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:55.634641 (+ 236us) server_negotiation.cc:299] Negotiation successful
0928 06:32:55.634708 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 06:32:55.667069 12270 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:32:55.667994 12270 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:32:55.671079 12270 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ff39fa7ce5f4c61ab3ad8ed59ad4a63: Generated new TSK 0
I20260928 06:32:55.671293 12270 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:32:55.679989 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 12188
W20260928 06:32:55.686841 12173 connection.cc:535] client connection to 127.0.0.1:35515 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35515): Connection reset by peer (error 104)
2026-09-28T06:32:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1475 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790576899143097-23430-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:32:55 dist-test-slave-d59m krb5kdc[12287](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:32:55 dist-test-slave-d59m krb5kdc[12287](info): set up 2 sockets
Sep 28 06:32:55 dist-test-slave-d59m krb5kdc[12287](info): commencing operation
krb5kdc: starting...
W20260928 06:32:56.773620 23430 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Sep 28 06:32:56 dist-test-slave-d59m krb5kdc[12287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:32:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:32:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790576899143097-23430-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790576899143097-23430-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:32:56.913738 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790576899143097-23430-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790576899143097-23430-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790576899143097-23430-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33035
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790576899143097-23430-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33035
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:32:57.021888 12300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:32:57.022099 12300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:32:57.022123 12300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:32:57.023303 12300 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:32:57.023344 12300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:32:57.023358 12300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:32:57.023372 12300 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:32:57.023386 12300 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:32:57.024931 12300 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790576899143097-23430-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790576899143097-23430-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33035
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790576899143097-23430-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12300
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790576899143097-23430-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:32:57.025202 12300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:32:57.025441 12300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:32:57.030292 12306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:57.030370 12308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:57.030417 12305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:32:57.030671 12300 server_base.cc:1034] running on GCE node
I20260928 06:32:57.030925 12300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:32:57.031278 12300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:32:57.032649 12300 hybrid_clock.cc:648] HybridClock initialized: now 1790577177032632 us; error 29 us; skew 500 ppm
Sep 28 06:32:57 dist-test-slave-d59m krb5kdc[12287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577177, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:32:57.035354 12300 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:32:57.035954 12300 webserver.cc:460] Webserver started at http://127.0.0.1:43827/ using document root <none> and password file <none>
I20260928 06:32:57.036199 12300 fs_manager.cc:359] Metadata directory not provided
I20260928 06:32:57.036270 12300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:32:57.036427 12300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:32:57.037601 12300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790576899143097-23430-0/minicluster-data/master-0/data/instance:
uuid: "e0bd0722a2274ff396d7049930ae597e"
format_stamp: "Formatted at 2026-09-28 06:32:57 on dist-test-slave-d59m"
server_key: "7dc99b1a0e5d932363282b72e304b9f4"
server_key_iv: "c0b8694db5fc6d61e1f3e016ca5b3761"
server_key_version: "encryptionkey@0"
I20260928 06:32:57.037976 12300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790576899143097-23430-0/minicluster-data/master-0/wal/instance:
uuid: "e0bd0722a2274ff396d7049930ae597e"
format_stamp: "Formatted at 2026-09-28 06:32:57 on dist-test-slave-d59m"
server_key: "7dc99b1a0e5d932363282b72e304b9f4"
server_key_iv: "c0b8694db5fc6d61e1f3e016ca5b3761"
server_key_version: "encryptionkey@0"
I20260928 06:32:57.039367 12300 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:32:57.040150 12315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:57.040383 12300 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:32:57.040501 12300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790576899143097-23430-0/minicluster-data/master-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790576899143097-23430-0/minicluster-data/master-0/wal
uuid: "e0bd0722a2274ff396d7049930ae597e"
format_stamp: "Formatted at 2026-09-28 06:32:57 on dist-test-slave-d59m"
server_key: "7dc99b1a0e5d932363282b72e304b9f4"
server_key_iv: "c0b8694db5fc6d61e1f3e016ca5b3761"
server_key_version: "encryptionkey@0"
I20260928 06:32:57.040601 12300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790576899143097-23430-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790576899143097-23430-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790576899143097-23430-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:32:57.064639 12300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:32:57.067368 12300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:32:57.067548 12300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:32:57.072604 12300 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33035
I20260928 06:32:57.072698 12367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33035 every 8 connection(s)
I20260928 06:32:57.073081 12300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
I20260928 06:32:57.074054 12368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:32:57.076428 12368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0bd0722a2274ff396d7049930ae597e: Bootstrap starting.
I20260928 06:32:57.077064 12368 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0bd0722a2274ff396d7049930ae597e: Neither blocks nor log segments found. Creating new log.
I20260928 06:32:57.077373 12368 log.cc:824] T 00000000000000000000000000000000 P e0bd0722a2274ff396d7049930ae597e: Log is configured to *not* fsync() on all Append() calls
I20260928 06:32:57.078127 12368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0bd0722a2274ff396d7049930ae597e: No bootstrap required, opened a new log
I20260928 06:32:57.079739 12368 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0bd0722a2274ff396d7049930ae597e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0bd0722a2274ff396d7049930ae597e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33035 } }
I20260928 06:32:57.079888 12368 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0bd0722a2274ff396d7049930ae597e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:32:57.079913 12368 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0bd0722a2274ff396d7049930ae597e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0bd0722a2274ff396d7049930ae597e, State: Initialized, Role: FOLLOWER
I20260928 06:32:57.080070 12368 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0bd0722a2274ff396d7049930ae597e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0bd0722a2274ff396d7049930ae597e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33035 } }
I20260928 06:32:57.080170 12368 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0bd0722a2274ff396d7049930ae597e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:32:57.080207 12368 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0bd0722a2274ff396d7049930ae597e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:32:57.080226 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 12300
I20260928 06:32:57.080259 12368 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0bd0722a2274ff396d7049930ae597e [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:32:57.080338 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790576899143097-23430-0/minicluster-data/master-0/wal/instance
I20260928 06:32:57.080541 23430 external_mini_cluster.cc:1428] Setting key 57e3b1302477b90949020158c92e93de
I20260928 06:32:57.081046 12368 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0bd0722a2274ff396d7049930ae597e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0bd0722a2274ff396d7049930ae597e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33035 } }
I20260928 06:32:57.081158 12368 leader_election.cc:302] T 00000000000000000000000000000000 P e0bd0722a2274ff396d7049930ae597e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0bd0722a2274ff396d7049930ae597e; no voters:
I20260928 06:32:57.081331 12368 leader_election.cc:288] T 00000000000000000000000000000000 P e0bd0722a2274ff396d7049930ae597e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:32:57.081522 12373 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0bd0722a2274ff396d7049930ae597e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:32:57.081626 12368 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0bd0722a2274ff396d7049930ae597e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:32:57.081642 12373 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0bd0722a2274ff396d7049930ae597e [term 1 LEADER]: Becoming Leader. State: Replica: e0bd0722a2274ff396d7049930ae597e, State: Running, Role: LEADER
I20260928 06:32:57.081924 12373 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0bd0722a2274ff396d7049930ae597e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0bd0722a2274ff396d7049930ae597e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33035 } }
I20260928 06:32:57.082263 12373 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0bd0722a2274ff396d7049930ae597e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0bd0722a2274ff396d7049930ae597e. Latest consensus state: current_term: 1 leader_uuid: "e0bd0722a2274ff396d7049930ae597e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0bd0722a2274ff396d7049930ae597e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33035 } } }
I20260928 06:32:57.082355 12373 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0bd0722a2274ff396d7049930ae597e [sys.catalog]: This master's current role is: LEADER
I20260928 06:32:57.082525 12374 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0bd0722a2274ff396d7049930ae597e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0bd0722a2274ff396d7049930ae597e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0bd0722a2274ff396d7049930ae597e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33035 } } }
I20260928 06:32:57.082594 12374 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0bd0722a2274ff396d7049930ae597e [sys.catalog]: This master's current role is: LEADER
I20260928 06:32:57.082705 12382 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:32:57.083264 12382 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:32:57.085070 12382 catalog_manager.cc:1347] Generated new cluster ID: 11e8c8fc4c3747c4a168f47bf9679e97
I20260928 06:32:57.085119 12382 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:32:57.090322 12372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:57.081114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.87:60729 (local address 127.0.0.1:33035)
0928 06:32:57.081337 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:57.081348 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:57.081432 (+ 84us) server_negotiation.cc:407] Connection header received
0928 06:32:57.081621 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:57.081630 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:57.081839 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:57.082329 (+ 490us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:32:57.082533 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:57.084360 (+ 1827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:57.085739 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:57.086355 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:57.086837 (+ 482us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:57.089350 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:32:57.089354 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:32:57.089354 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:32:57.089595 (+ 241us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:32:57.089638 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:32:57.089655 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:32:57.089996 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:57.090046 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 06:32:57.090079 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":86,"threads_started":1}
I20260928 06:32:57.092069 23430 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:32:57.097885 12372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:57.093691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.87:48407 (local address 127.0.0.1:33035)
0928 06:32:57.093977 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:57.093979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:57.094087 (+ 108us) server_negotiation.cc:407] Connection header received
0928 06:32:57.094180 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:57.094181 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:57.094193 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:57.094224 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:32:57.094442 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:57.095247 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:57.096339 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:57.096681 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:57.096725 (+ 44us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:57.097588 (+ 863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:32:57.097589 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:32:57.097590 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:32:57.097595 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:32:57.097609 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:32:57.097614 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:32:57.097647 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:57.097781 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 06:32:57.097795 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263}
I20260928 06:32:57.099538 12382 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:32:57.100255 12382 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:32:57.110535 12382 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0bd0722a2274ff396d7049930ae597e: Generated new TSK 0
I20260928 06:32:57.110771 12382 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:32:57.116145 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 12300
W20260928 06:32:57.121388 12285 connection.cc:535] client connection to 127.0.0.1:33035 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33035): Connection reset by peer (error 104)
2026-09-28T06:32:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1433 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790576899143097-23430-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:32:57 dist-test-slave-d59m krb5kdc[12399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:32:57 dist-test-slave-d59m krb5kdc[12399](info): set up 2 sockets
Sep 28 06:32:57 dist-test-slave-d59m krb5kdc[12399](info): commencing operation
krb5kdc: starting...
W20260928 06:32:58.241276 23430 mini_kdc.cc:121] Time spent starting KDC: real 1.106s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Sep 28 06:32:58 dist-test-slave-d59m krb5kdc[12399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577178, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:32:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:32:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790576899143097-23430-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790576899143097-23430-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:32:58.370878 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790576899143097-23430-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790576899143097-23430-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790576899143097-23430-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45517
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790576899143097-23430-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45517
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:32:58.461871 12412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:32:58.462096 12412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:32:58.462121 12412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:32:58.463325 12412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:32:58.463371 12412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:32:58.463385 12412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:32:58.463398 12412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:32:58.463410 12412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:32:58.465051 12412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790576899143097-23430-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790576899143097-23430-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45517
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790576899143097-23430-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790576899143097-23430-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:32:58.465325 12412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:32:58.465575 12412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:32:58.470530 12417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:32:58.470587 12412 server_base.cc:1034] running on GCE node
W20260928 06:32:58.470717 12418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:58.470525 12420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:32:58.470983 12412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:32:58.471285 12412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:32:58.472473 12412 hybrid_clock.cc:648] HybridClock initialized: now 1790577178472457 us; error 34 us; skew 500 ppm
Sep 28 06:32:58 dist-test-slave-d59m krb5kdc[12399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577178, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:32:58.475075 12412 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:32:58.475636 12412 webserver.cc:460] Webserver started at http://127.0.0.1:41427/ using document root <none> and password file <none>
I20260928 06:32:58.475898 12412 fs_manager.cc:359] Metadata directory not provided
I20260928 06:32:58.475975 12412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:32:58.476135 12412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:32:58.477332 12412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790576899143097-23430-0/minicluster-data/master-0/data/instance:
uuid: "9d426efd433a449098e406114a7e92ee"
format_stamp: "Formatted at 2026-09-28 06:32:58 on dist-test-slave-d59m"
server_key: "414d655fa1821489b5a5e86d615a485a"
server_key_iv: "affe2d1aa5e05469dc1590fcbe2d64e5"
server_key_version: "encryptionkey@0"
I20260928 06:32:58.477741 12412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790576899143097-23430-0/minicluster-data/master-0/wal/instance:
uuid: "9d426efd433a449098e406114a7e92ee"
format_stamp: "Formatted at 2026-09-28 06:32:58 on dist-test-slave-d59m"
server_key: "414d655fa1821489b5a5e86d615a485a"
server_key_iv: "affe2d1aa5e05469dc1590fcbe2d64e5"
server_key_version: "encryptionkey@0"
I20260928 06:32:58.479219 12412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:32:58.480037 12427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:58.480234 12412 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:32:58.480345 12412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790576899143097-23430-0/minicluster-data/master-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790576899143097-23430-0/minicluster-data/master-0/wal
uuid: "9d426efd433a449098e406114a7e92ee"
format_stamp: "Formatted at 2026-09-28 06:32:58 on dist-test-slave-d59m"
server_key: "414d655fa1821489b5a5e86d615a485a"
server_key_iv: "affe2d1aa5e05469dc1590fcbe2d64e5"
server_key_version: "encryptionkey@0"
I20260928 06:32:58.480425 12412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790576899143097-23430-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790576899143097-23430-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790576899143097-23430-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:32:58.483904 12412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:32:58.486330 12412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:32:58.486624 12412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:32:58.490988 12412 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45517
I20260928 06:32:58.491034 12479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45517 every 8 connection(s)
I20260928 06:32:58.491395 12412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
I20260928 06:32:58.492669 12480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:32:58.495857 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 12412
I20260928 06:32:58.495986 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790576899143097-23430-0/minicluster-data/master-0/wal/instance
I20260928 06:32:58.496172 23430 external_mini_cluster.cc:1428] Setting key 6b674f758ba83ea39f8fc2474b706270
I20260928 06:32:58.497274 12480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d426efd433a449098e406114a7e92ee: Bootstrap starting.
I20260928 06:32:58.498031 12480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d426efd433a449098e406114a7e92ee: Neither blocks nor log segments found. Creating new log.
I20260928 06:32:58.498379 12480 log.cc:824] T 00000000000000000000000000000000 P 9d426efd433a449098e406114a7e92ee: Log is configured to *not* fsync() on all Append() calls
I20260928 06:32:58.499215 12480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d426efd433a449098e406114a7e92ee: No bootstrap required, opened a new log
I20260928 06:32:58.499310 12483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:58.496816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.87:48569 (local address 127.0.0.1:45517)
0928 06:32:58.497025 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:58.497036 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:58.497069 (+ 33us) server_negotiation.cc:407] Connection header received
0928 06:32:58.497639 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:58.497650 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:58.497692 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:58.497859 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:32:58.498708 (+ 849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:32:58.498712 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:32:58.498712 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:32:58.498717 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:32:58.498759 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:32:58.498774 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:32:58.498871 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:58.498947 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 06:32:58.499096 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":106,"threads_started":1}
I20260928 06:32:58.500415 23430 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:32:58.501960 12480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d426efd433a449098e406114a7e92ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d426efd433a449098e406114a7e92ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45517 } }
I20260928 06:32:58.502152 12480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d426efd433a449098e406114a7e92ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:32:58.502198 12480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d426efd433a449098e406114a7e92ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d426efd433a449098e406114a7e92ee, State: Initialized, Role: FOLLOWER
I20260928 06:32:58.502323 12480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d426efd433a449098e406114a7e92ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d426efd433a449098e406114a7e92ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45517 } }
I20260928 06:32:58.502435 12480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d426efd433a449098e406114a7e92ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:32:58.502481 12480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d426efd433a449098e406114a7e92ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:32:58.502532 12480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d426efd433a449098e406114a7e92ee [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:32:58.503094 12480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d426efd433a449098e406114a7e92ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d426efd433a449098e406114a7e92ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45517 } }
I20260928 06:32:58.503217 12480 leader_election.cc:302] T 00000000000000000000000000000000 P 9d426efd433a449098e406114a7e92ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d426efd433a449098e406114a7e92ee; no voters:
I20260928 06:32:58.503377 12480 leader_election.cc:288] T 00000000000000000000000000000000 P 9d426efd433a449098e406114a7e92ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:32:58.503399 12490 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d426efd433a449098e406114a7e92ee [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:32:58.503518 12490 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d426efd433a449098e406114a7e92ee [term 1 LEADER]: Becoming Leader. State: Replica: 9d426efd433a449098e406114a7e92ee, State: Running, Role: LEADER
I20260928 06:32:58.503599 12490 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d426efd433a449098e406114a7e92ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d426efd433a449098e406114a7e92ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45517 } }
I20260928 06:32:58.503639 12480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d426efd433a449098e406114a7e92ee [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:32:58.503944 12490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d426efd433a449098e406114a7e92ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d426efd433a449098e406114a7e92ee. Latest consensus state: current_term: 1 leader_uuid: "9d426efd433a449098e406114a7e92ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d426efd433a449098e406114a7e92ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45517 } } }
I20260928 06:32:58.504055 12490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d426efd433a449098e406114a7e92ee [sys.catalog]: This master's current role is: LEADER
I20260928 06:32:58.503958 12491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d426efd433a449098e406114a7e92ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d426efd433a449098e406114a7e92ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d426efd433a449098e406114a7e92ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45517 } } }
I20260928 06:32:58.504243 12491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d426efd433a449098e406114a7e92ee [sys.catalog]: This master's current role is: LEADER
I20260928 06:32:58.504293 12496 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:32:58.504954 12496 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:32:58.506726 12496 catalog_manager.cc:1347] Generated new cluster ID: 7b6f79abbb3949faba61ed6119222c44
I20260928 06:32:58.506775 12496 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:32:58.509706 12483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:58.502148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.87:36821 (local address 127.0.0.1:45517)
0928 06:32:58.502282 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:58.502284 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:58.505191 (+ 2907us) server_negotiation.cc:407] Connection header received
0928 06:32:58.505305 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:58.505306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:58.505312 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:58.505376 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:32:58.509254 (+ 3878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:32:58.509256 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:32:58.509256 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:32:58.509262 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:32:58.509280 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:32:58.509286 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:32:58.509335 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:58.509585 (+ 250us) server_negotiation.cc:299] Negotiation successful
0928 06:32:58.509601 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260928 06:32:58.528029 12496 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:32:58.528270 12496 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:32:58.539007 12496 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d426efd433a449098e406114a7e92ee: Generated new TSK 0
I20260928 06:32:58.539284 12496 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:32:58.541960 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 12412
2026-09-28T06:32:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1425 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790576899143097-23430-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:32:58 dist-test-slave-d59m krb5kdc[12511](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:32:58 dist-test-slave-d59m krb5kdc[12511](info): set up 2 sockets
Sep 28 06:32:58 dist-test-slave-d59m krb5kdc[12511](info): commencing operation
krb5kdc: starting...
W20260928 06:32:59.597216 23430 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Sep 28 06:32:59 dist-test-slave-d59m krb5kdc[12511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:32:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:32:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab.
Entry for principal kudu/127.22.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab.
I20260928 06:32:59.723623 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790576899143097-23430-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790576899143097-23430-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790576899143097-23430-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:43207
--webserver_interface=127.22.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab
--principal=kudu/127.22.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:40039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.225.190:43207
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:32:59.836017 12524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:32:59.836303 12524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:32:59.836335 12524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:32:59.837944 12524 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:32:59.838012 12524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:32:59.838032 12524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:32:59.838049 12524 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:32:59.838066 12524 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:32:59.840075 12524 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:40039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790576899143097-23430-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790576899143097-23430-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.225.190:43207
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab
--principal=kudu/127.22.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:43207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790576899143097-23430-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:32:59.840327 12524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:32:59.840615 12524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:32:59.845757 12529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:59.845767 12530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:59.845799 12532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:32:59.846213 12524 server_base.cc:1034] running on GCE node
I20260928 06:32:59.846482 12524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:32:59.846784 12524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:32:59.847954 12524 hybrid_clock.cc:648] HybridClock initialized: now 1790577179847929 us; error 38 us; skew 500 ppm
Sep 28 06:32:59 dist-test-slave-d59m krb5kdc[12511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577179, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:32:59.850876 12524 init.cc:377] Logged in from keytab as kudu/127.22.225.190@KRBTEST.COM (short username kudu)
I20260928 06:32:59.851320 12524 webserver.cc:460] Webserver started at http://127.22.225.190:36577/ using document root <none> and password file <none>
I20260928 06:32:59.851565 12524 fs_manager.cc:359] Metadata directory not provided
I20260928 06:32:59.851634 12524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:32:59.851779 12524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:32:59.852944 12524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790576899143097-23430-0/minicluster-data/master-0/data/instance:
uuid: "768073bf055b44bcbd4764c94d63a364"
format_stamp: "Formatted at 2026-09-28 06:32:59 on dist-test-slave-d59m"
server_key: "7cb11d98196b17436b4fa0c7322d46be"
server_key_iv: "54e4dc6307d782e3959a2895b4f99ad4"
server_key_version: "encryptionkey@0"
I20260928 06:32:59.853305 12524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790576899143097-23430-0/minicluster-data/master-0/wal/instance:
uuid: "768073bf055b44bcbd4764c94d63a364"
format_stamp: "Formatted at 2026-09-28 06:32:59 on dist-test-slave-d59m"
server_key: "7cb11d98196b17436b4fa0c7322d46be"
server_key_iv: "54e4dc6307d782e3959a2895b4f99ad4"
server_key_version: "encryptionkey@0"
I20260928 06:32:59.854664 12524 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:32:59.855428 12539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:59.855602 12524 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:32:59.855700 12524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790576899143097-23430-0/minicluster-data/master-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790576899143097-23430-0/minicluster-data/master-0/wal
uuid: "768073bf055b44bcbd4764c94d63a364"
format_stamp: "Formatted at 2026-09-28 06:32:59 on dist-test-slave-d59m"
server_key: "7cb11d98196b17436b4fa0c7322d46be"
server_key_iv: "54e4dc6307d782e3959a2895b4f99ad4"
server_key_version: "encryptionkey@0"
I20260928 06:32:59.855782 12524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790576899143097-23430-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790576899143097-23430-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790576899143097-23430-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:32:59.871803 12524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:32:59.874260 12524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:32:59.874420 12524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:32:59.878734 12524 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.190:43207
I20260928 06:32:59.878786 12591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.190:43207 every 8 connection(s)
I20260928 06:32:59.879141 12524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
I20260928 06:32:59.879443 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 12524
I20260928 06:32:59.879563 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790576899143097-23430-0/minicluster-data/master-0/wal/instance
I20260928 06:32:59.879765 23430 external_mini_cluster.cc:1428] Setting key 569b37b233413d6941658aed18076c94
I20260928 06:32:59.880199 12592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:32:59.883412 12592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 768073bf055b44bcbd4764c94d63a364: Bootstrap starting.
I20260928 06:32:59.884032 12592 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 768073bf055b44bcbd4764c94d63a364: Neither blocks nor log segments found. Creating new log.
I20260928 06:32:59.884327 12592 log.cc:824] T 00000000000000000000000000000000 P 768073bf055b44bcbd4764c94d63a364: Log is configured to *not* fsync() on all Append() calls
I20260928 06:32:59.885135 12592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 768073bf055b44bcbd4764c94d63a364: No bootstrap required, opened a new log
I20260928 06:32:59.887404 12592 raft_consensus.cc:348] T 00000000000000000000000000000000 P 768073bf055b44bcbd4764c94d63a364 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "768073bf055b44bcbd4764c94d63a364" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 43207 } }
I20260928 06:32:59.887588 12592 raft_consensus.cc:374] T 00000000000000000000000000000000 P 768073bf055b44bcbd4764c94d63a364 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
Sep 28 06:32:59 dist-test-slave-d59m krb5kdc[12511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:32:59.887670 12592 raft_consensus.cc:729] T 00000000000000000000000000000000 P 768073bf055b44bcbd4764c94d63a364 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 768073bf055b44bcbd4764c94d63a364, State: Initialized, Role: FOLLOWER
I20260928 06:32:59.887823 12592 consensus_queue.cc:260] T 00000000000000000000000000000000 P 768073bf055b44bcbd4764c94d63a364 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "768073bf055b44bcbd4764c94d63a364" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 43207 } }
I20260928 06:32:59.887930 12592 raft_consensus.cc:388] T 00000000000000000000000000000000 P 768073bf055b44bcbd4764c94d63a364 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:32:59.887995 12592 raft_consensus.cc:482] T 00000000000000000000000000000000 P 768073bf055b44bcbd4764c94d63a364 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:32:59.888055 12592 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 768073bf055b44bcbd4764c94d63a364 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:32:59.888703 12592 raft_consensus.cc:504] T 00000000000000000000000000000000 P 768073bf055b44bcbd4764c94d63a364 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "768073bf055b44bcbd4764c94d63a364" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 43207 } }
I20260928 06:32:59.888887 12592 leader_election.cc:302] T 00000000000000000000000000000000 P 768073bf055b44bcbd4764c94d63a364 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 768073bf055b44bcbd4764c94d63a364; no voters:
I20260928 06:32:59.889093 12592 leader_election.cc:288] T 00000000000000000000000000000000 P 768073bf055b44bcbd4764c94d63a364 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:32:59.889133 12597 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 768073bf055b44bcbd4764c94d63a364 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:32:59.889433 12592 sys_catalog.cc:564] T 00000000000000000000000000000000 P 768073bf055b44bcbd4764c94d63a364 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:32:59.889571 12597 raft_consensus.cc:686] T 00000000000000000000000000000000 P 768073bf055b44bcbd4764c94d63a364 [term 1 LEADER]: Becoming Leader. State: Replica: 768073bf055b44bcbd4764c94d63a364, State: Running, Role: LEADER
I20260928 06:32:59.889683 12597 consensus_queue.cc:237] T 00000000000000000000000000000000 P 768073bf055b44bcbd4764c94d63a364 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "768073bf055b44bcbd4764c94d63a364" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 43207 } }
I20260928 06:32:59.890116 12599 sys_catalog.cc:455] T 00000000000000000000000000000000 P 768073bf055b44bcbd4764c94d63a364 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 768073bf055b44bcbd4764c94d63a364. Latest consensus state: current_term: 1 leader_uuid: "768073bf055b44bcbd4764c94d63a364" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "768073bf055b44bcbd4764c94d63a364" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 43207 } } }
I20260928 06:32:59.890246 12599 sys_catalog.cc:458] T 00000000000000000000000000000000 P 768073bf055b44bcbd4764c94d63a364 [sys.catalog]: This master's current role is: LEADER
I20260928 06:32:59.890815 12598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 768073bf055b44bcbd4764c94d63a364 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "768073bf055b44bcbd4764c94d63a364" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "768073bf055b44bcbd4764c94d63a364" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 43207 } } }
I20260928 06:32:59.890956 12598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 768073bf055b44bcbd4764c94d63a364 [sys.catalog]: This master's current role is: LEADER
W20260928 06:32:59.891516 12613 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 768073bf055b44bcbd4764c94d63a364: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:32:59.892020 12613 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:32:59.892380 12595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:59.880259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43713 (local address 127.22.225.190:43207)
0928 06:32:59.880461 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:59.880470 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:59.880501 (+ 31us) server_negotiation.cc:407] Connection header received
0928 06:32:59.881064 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:59.881073 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:59.881144 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:59.881297 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:32:59.882215 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:59.883501 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:59.884487 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:59.885119 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:59.885248 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:59.888227 (+ 2979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:32:59.888232 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:32:59.888232 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:32:59.888245 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:32:59.890295 (+ 2050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:32:59.890797 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:32:59.890800 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:32:59.890801 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:32:59.890964 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:32:59.891330 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:32:59.891331 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:32:59.891331 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:32:59.891616 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:32:59.891797 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:59.892149 (+ 352us) server_negotiation.cc:299] Negotiation successful
0928 06:32:59.892239 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":98,"threads_started":1}
I20260928 06:32:59.892499 12606 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:32:59.892838 12606 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:32:59.893510 23430 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:32:59.894452 12606 catalog_manager.cc:1347] Generated new cluster ID: de709a8395584b328cf601f291e0f95d
I20260928 06:32:59.894510 12606 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:32:59.903743 12595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:59.894997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53141 (local address 127.22.225.190:43207)
0928 06:32:59.896673 (+ 1676us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:59.896675 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:59.896683 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:32:59.896702 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:59.896703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:59.896712 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:59.896761 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:32:59.897412 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:59.898150 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:59.899124 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:59.899436 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:59.899472 (+ 36us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:59.901043 (+ 1571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:32:59.901044 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:32:59.901045 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:32:59.901051 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:32:59.902572 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:32:59.902947 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:32:59.902948 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:32:59.902949 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:32:59.902994 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:32:59.903201 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:32:59.903202 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:32:59.903202 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:32:59.903310 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:32:59.903378 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:59.903578 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 06:32:59.903637 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1657}
I20260928 06:32:59.908968 12606 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:32:59.909499 12606 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:32:59.918169 12606 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 768073bf055b44bcbd4764c94d63a364: Generated new TSK 0
I20260928 06:32:59.918401 12606 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:32:59.928207 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 12524
W20260928 06:32:59.932871 12509 connection.cc:535] client connection to 127.22.225.190:43207 recv error: Network error: failed to read from TLS socket (remote: 127.22.225.190:43207): Connection reset by peer (error 104)
2026-09-28T06:32:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1385 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790576899143097-23430-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:32:59 dist-test-slave-d59m krb5kdc[12623](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:32:59 dist-test-slave-d59m krb5kdc[12623](info): set up 2 sockets
Sep 28 06:32:59 dist-test-slave-d59m krb5kdc[12623](info): commencing operation
krb5kdc: starting...
W20260928 06:33:01.001389 23430 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Sep 28 06:33:01 dist-test-slave-d59m krb5kdc[12623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:33:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:33:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab.
Entry for principal kudu/127.22.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab.
I20260928 06:33:01.128008 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790576899143097-23430-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790576899143097-23430-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790576899143097-23430-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:44867
--webserver_interface=127.22.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab
--principal=kudu/127.22.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:37137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.225.190:44867
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:01.211467 12636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:01.211722 12636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:01.211766 12636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:01.212980 12636 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:33:01.213052 12636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:01.213074 12636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:01.213097 12636 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:33:01.213127 12636 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:33:01.214958 12636 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:37137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790576899143097-23430-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790576899143097-23430-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.225.190:44867
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab
--principal=kudu/127.22.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:44867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12636
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790576899143097-23430-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:01.215296 12636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:01.215569 12636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:01.221292 12642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:01.221261 12641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:01.221252 12644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:01.221905 12636 server_base.cc:1034] running on GCE node
I20260928 06:33:01.222178 12636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:01.222522 12636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:01.223685 12636 hybrid_clock.cc:648] HybridClock initialized: now 1790577181223673 us; error 31 us; skew 500 ppm
Sep 28 06:33:01 dist-test-slave-d59m krb5kdc[12623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577181, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:01.231832 12636 init.cc:377] Logged in from keytab as kudu/127.22.225.190@KRBTEST.COM (short username kudu)
I20260928 06:33:01.232372 12636 webserver.cc:460] Webserver started at http://127.22.225.190:34589/ using document root <none> and password file <none>
I20260928 06:33:01.232728 12636 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:01.232802 12636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:01.232977 12636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:01.234102 12636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790576899143097-23430-0/minicluster-data/master-0/data/instance:
uuid: "eba39615e05245f98325f4f9a4ba878c"
format_stamp: "Formatted at 2026-09-28 06:33:01 on dist-test-slave-d59m"
server_key: "29ab26f46ca14b0a4e589e5b7d84e2f2"
server_key_iv: "26d415e0ae2438470ea4ef09bcaae243"
server_key_version: "encryptionkey@0"
I20260928 06:33:01.234493 12636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790576899143097-23430-0/minicluster-data/master-0/wal/instance:
uuid: "eba39615e05245f98325f4f9a4ba878c"
format_stamp: "Formatted at 2026-09-28 06:33:01 on dist-test-slave-d59m"
server_key: "29ab26f46ca14b0a4e589e5b7d84e2f2"
server_key_iv: "26d415e0ae2438470ea4ef09bcaae243"
server_key_version: "encryptionkey@0"
I20260928 06:33:01.235922 12636 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:33:01.236776 12651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:01.236927 12636 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:01.237023 12636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790576899143097-23430-0/minicluster-data/master-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790576899143097-23430-0/minicluster-data/master-0/wal
uuid: "eba39615e05245f98325f4f9a4ba878c"
format_stamp: "Formatted at 2026-09-28 06:33:01 on dist-test-slave-d59m"
server_key: "29ab26f46ca14b0a4e589e5b7d84e2f2"
server_key_iv: "26d415e0ae2438470ea4ef09bcaae243"
server_key_version: "encryptionkey@0"
I20260928 06:33:01.237100 12636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790576899143097-23430-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790576899143097-23430-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790576899143097-23430-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:01.250589 12636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:01.253082 12636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:01.253266 12636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:01.257984 12636 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.190:44867
I20260928 06:33:01.258011 12703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.190:44867 every 8 connection(s)
I20260928 06:33:01.258379 12636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
I20260928 06:33:01.259279 12704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:01.261734 12704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eba39615e05245f98325f4f9a4ba878c: Bootstrap starting.
I20260928 06:33:01.262346 12704 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eba39615e05245f98325f4f9a4ba878c: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:01.262642 12704 log.cc:824] T 00000000000000000000000000000000 P eba39615e05245f98325f4f9a4ba878c: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:01.263439 12704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eba39615e05245f98325f4f9a4ba878c: No bootstrap required, opened a new log
I20260928 06:33:01.263487 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 12636
I20260928 06:33:01.263607 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790576899143097-23430-0/minicluster-data/master-0/wal/instance
I20260928 06:33:01.263814 23430 external_mini_cluster.cc:1428] Setting key 03810cde468b61206472b47157aec8d8
I20260928 06:33:01.265544 12704 raft_consensus.cc:348] T 00000000000000000000000000000000 P eba39615e05245f98325f4f9a4ba878c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eba39615e05245f98325f4f9a4ba878c" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 44867 } }
I20260928 06:33:01.265709 12704 raft_consensus.cc:374] T 00000000000000000000000000000000 P eba39615e05245f98325f4f9a4ba878c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:01.265772 12704 raft_consensus.cc:729] T 00000000000000000000000000000000 P eba39615e05245f98325f4f9a4ba878c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eba39615e05245f98325f4f9a4ba878c, State: Initialized, Role: FOLLOWER
I20260928 06:33:01.265944 12704 consensus_queue.cc:260] T 00000000000000000000000000000000 P eba39615e05245f98325f4f9a4ba878c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eba39615e05245f98325f4f9a4ba878c" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 44867 } }
I20260928 06:33:01.266039 12704 raft_consensus.cc:388] T 00000000000000000000000000000000 P eba39615e05245f98325f4f9a4ba878c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:33:01.266098 12704 raft_consensus.cc:482] T 00000000000000000000000000000000 P eba39615e05245f98325f4f9a4ba878c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:33:01.266149 12704 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eba39615e05245f98325f4f9a4ba878c [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:01.266700 12704 raft_consensus.cc:504] T 00000000000000000000000000000000 P eba39615e05245f98325f4f9a4ba878c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eba39615e05245f98325f4f9a4ba878c" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 44867 } }
I20260928 06:33:01.266817 12704 leader_election.cc:302] T 00000000000000000000000000000000 P eba39615e05245f98325f4f9a4ba878c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eba39615e05245f98325f4f9a4ba878c; no voters:
I20260928 06:33:01.267005 12704 leader_election.cc:288] T 00000000000000000000000000000000 P eba39615e05245f98325f4f9a4ba878c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:33:01.267025 12709 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eba39615e05245f98325f4f9a4ba878c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:33:01.267225 12709 raft_consensus.cc:686] T 00000000000000000000000000000000 P eba39615e05245f98325f4f9a4ba878c [term 1 LEADER]: Becoming Leader. State: Replica: eba39615e05245f98325f4f9a4ba878c, State: Running, Role: LEADER
I20260928 06:33:01.267351 12704 sys_catalog.cc:564] T 00000000000000000000000000000000 P eba39615e05245f98325f4f9a4ba878c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:33:01.267349 12709 consensus_queue.cc:237] T 00000000000000000000000000000000 P eba39615e05245f98325f4f9a4ba878c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eba39615e05245f98325f4f9a4ba878c" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 44867 } }
I20260928 06:33:01.267787 12709 sys_catalog.cc:455] T 00000000000000000000000000000000 P eba39615e05245f98325f4f9a4ba878c [sys.catalog]: SysCatalogTable state changed. Reason: New leader eba39615e05245f98325f4f9a4ba878c. Latest consensus state: current_term: 1 leader_uuid: "eba39615e05245f98325f4f9a4ba878c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eba39615e05245f98325f4f9a4ba878c" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 44867 } } }
I20260928 06:33:01.267925 12709 sys_catalog.cc:458] T 00000000000000000000000000000000 P eba39615e05245f98325f4f9a4ba878c [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:01.268141 12711 sys_catalog.cc:455] T 00000000000000000000000000000000 P eba39615e05245f98325f4f9a4ba878c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eba39615e05245f98325f4f9a4ba878c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eba39615e05245f98325f4f9a4ba878c" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 44867 } } }
I20260928 06:33:01.268262 12711 sys_catalog.cc:458] T 00000000000000000000000000000000 P eba39615e05245f98325f4f9a4ba878c [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:01.268420 12717 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:33:01.269052 12717 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:33:01.270907 12717 catalog_manager.cc:1347] Generated new cluster ID: 610786cf6ed7422ebe8a2ceeb0465d77
I20260928 06:33:01.270958 12717 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:33:01.271224 12708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:01.264354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58831 (local address 127.22.225.190:44867)
0928 06:33:01.264592 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:01.264603 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:01.264635 (+ 32us) server_negotiation.cc:407] Connection header received
0928 06:33:01.265532 (+ 897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:01.265542 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:01.265607 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:01.265756 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:01.266099 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:01.267673 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:01.268736 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:01.269302 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:01.269425 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:01.270402 (+ 977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:01.270407 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:01.270407 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:33:01.270642 (+ 235us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:01.270680 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:33:01.270696 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:01.270780 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:01.270985 (+ 205us) server_negotiation.cc:299] Negotiation successful
0928 06:33:01.271015 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":101,"threads_started":1}
I20260928 06:33:01.275954 12717 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:33:01.276374 23430 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:33:01.276710 12717 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:33:01.284992 12717 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eba39615e05245f98325f4f9a4ba878c: Generated new TSK 0
I20260928 06:33:01.285058 12708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:01.280990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38807 (local address 127.22.225.190:44867)
0928 06:33:01.281031 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:01.281033 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:01.281321 (+ 288us) server_negotiation.cc:407] Connection header received
0928 06:33:01.281391 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:01.281391 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:01.281402 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:01.281455 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:01.281643 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:01.282405 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:01.283429 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:01.283756 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:01.283797 (+ 41us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:01.284696 (+ 899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:01.284698 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:01.284698 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:33:01.284704 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:01.284722 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:33:01.284728 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:01.284764 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:01.284907 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 06:33:01.284923 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:33:01.285264 12717 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:33:01.287415 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 12636
W20260928 06:33:01.292861 12621 connection.cc:535] client connection to 127.22.225.190:44867 recv error: Network error: failed to read from TLS socket (remote: 127.22.225.190:44867): Connection reset by peer (error 104)
2026-09-28T06:33:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1360 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790576899143097-23430-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:33:01 dist-test-slave-d59m krb5kdc[12736](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:33:01 dist-test-slave-d59m krb5kdc[12736](info): set up 2 sockets
Sep 28 06:33:01 dist-test-slave-d59m krb5kdc[12736](info): commencing operation
krb5kdc: starting...
W20260928 06:33:02.411273 23430 mini_kdc.cc:121] Time spent starting KDC: real 1.104s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Sep 28 06:33:02 dist-test-slave-d59m krb5kdc[12736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:33:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:33:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab.
Entry for principal kudu/127.22.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab.
I20260928 06:33:02.551623 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790576899143097-23430-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790576899143097-23430-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790576899143097-23430-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:39841
--webserver_interface=127.22.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab
--principal=kudu/127.22.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:35815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.225.190:39841
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:02.648713 12749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:02.648980 12749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:02.649006 12749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:02.650187 12749 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:33:02.650233 12749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:02.650247 12749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:02.650259 12749 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:33:02.650271 12749 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:33:02.651794 12749 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:35815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790576899143097-23430-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790576899143097-23430-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.225.190:39841
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab
--principal=kudu/127.22.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:39841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12749
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790576899143097-23430-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:02.652056 12749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:02.652289 12749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:02.657186 12755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:02.657178 12754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:02.657181 12757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:02.657630 12749 server_base.cc:1034] running on GCE node
I20260928 06:33:02.657866 12749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:02.658149 12749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:02.659313 12749 hybrid_clock.cc:648] HybridClock initialized: now 1790577182659300 us; error 31 us; skew 500 ppm
Sep 28 06:33:02 dist-test-slave-d59m krb5kdc[12736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577182, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:02.662073 12749 init.cc:377] Logged in from keytab as kudu/127.22.225.190@KRBTEST.COM (short username kudu)
I20260928 06:33:02.662560 12749 webserver.cc:460] Webserver started at http://127.22.225.190:44511/ using document root <none> and password file <none>
I20260928 06:33:02.662781 12749 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:02.662849 12749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:02.662997 12749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:02.664075 12749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790576899143097-23430-0/minicluster-data/master-0/data/instance:
uuid: "f8d74ace8609422dae2f2038ff8b5621"
format_stamp: "Formatted at 2026-09-28 06:33:02 on dist-test-slave-d59m"
server_key: "92e4a81374e81a8d5de4c8f5298f8154"
server_key_iv: "722c6e12654948ea896d76e64c895735"
server_key_version: "encryptionkey@0"
I20260928 06:33:02.664439 12749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790576899143097-23430-0/minicluster-data/master-0/wal/instance:
uuid: "f8d74ace8609422dae2f2038ff8b5621"
format_stamp: "Formatted at 2026-09-28 06:33:02 on dist-test-slave-d59m"
server_key: "92e4a81374e81a8d5de4c8f5298f8154"
server_key_iv: "722c6e12654948ea896d76e64c895735"
server_key_version: "encryptionkey@0"
I20260928 06:33:02.665870 12749 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:33:02.666676 12764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:02.666913 12749 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:33:02.667022 12749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790576899143097-23430-0/minicluster-data/master-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790576899143097-23430-0/minicluster-data/master-0/wal
uuid: "f8d74ace8609422dae2f2038ff8b5621"
format_stamp: "Formatted at 2026-09-28 06:33:02 on dist-test-slave-d59m"
server_key: "92e4a81374e81a8d5de4c8f5298f8154"
server_key_iv: "722c6e12654948ea896d76e64c895735"
server_key_version: "encryptionkey@0"
I20260928 06:33:02.667109 12749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790576899143097-23430-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790576899143097-23430-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790576899143097-23430-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:02.670444 12749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:02.672688 12749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:02.672835 12749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:02.677083 12749 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.190:39841
I20260928 06:33:02.677132 12816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.190:39841 every 8 connection(s)
I20260928 06:33:02.677459 12749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
I20260928 06:33:02.678642 12817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:02.681787 12817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8d74ace8609422dae2f2038ff8b5621: Bootstrap starting.
I20260928 06:33:02.682521 12817 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8d74ace8609422dae2f2038ff8b5621: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:02.682828 12817 log.cc:824] T 00000000000000000000000000000000 P f8d74ace8609422dae2f2038ff8b5621: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:02.683627 12817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8d74ace8609422dae2f2038ff8b5621: No bootstrap required, opened a new log
I20260928 06:33:02.685904 12817 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8d74ace8609422dae2f2038ff8b5621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8d74ace8609422dae2f2038ff8b5621" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 39841 } }
I20260928 06:33:02.686090 12817 raft_consensus.cc:374] T 00000000000000000000000000000000 P f8d74ace8609422dae2f2038ff8b5621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:02.686123 12817 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8d74ace8609422dae2f2038ff8b5621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8d74ace8609422dae2f2038ff8b5621, State: Initialized, Role: FOLLOWER
I20260928 06:33:02.686260 12817 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8d74ace8609422dae2f2038ff8b5621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8d74ace8609422dae2f2038ff8b5621" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 39841 } }
I20260928 06:33:02.686329 12817 raft_consensus.cc:388] T 00000000000000000000000000000000 P f8d74ace8609422dae2f2038ff8b5621 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:33:02.686357 12817 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8d74ace8609422dae2f2038ff8b5621 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:33:02.686395 12817 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8d74ace8609422dae2f2038ff8b5621 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:02.687186 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 12749
I20260928 06:33:02.687312 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790576899143097-23430-0/minicluster-data/master-0/wal/instance
I20260928 06:33:02.687454 12817 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8d74ace8609422dae2f2038ff8b5621 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8d74ace8609422dae2f2038ff8b5621" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 39841 } }
I20260928 06:33:02.687510 23430 external_mini_cluster.cc:1428] Setting key b8ce82395ec230a777cee2df03a5ab7e
I20260928 06:33:02.687563 12817 leader_election.cc:302] T 00000000000000000000000000000000 P f8d74ace8609422dae2f2038ff8b5621 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f8d74ace8609422dae2f2038ff8b5621; no voters:
I20260928 06:33:02.687719 12817 leader_election.cc:288] T 00000000000000000000000000000000 P f8d74ace8609422dae2f2038ff8b5621 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:33:02.687773 12820 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f8d74ace8609422dae2f2038ff8b5621 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:33:02.687994 12820 raft_consensus.cc:686] T 00000000000000000000000000000000 P f8d74ace8609422dae2f2038ff8b5621 [term 1 LEADER]: Becoming Leader. State: Replica: f8d74ace8609422dae2f2038ff8b5621, State: Running, Role: LEADER
I20260928 06:33:02.688031 12817 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8d74ace8609422dae2f2038ff8b5621 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:33:02.688148 12820 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8d74ace8609422dae2f2038ff8b5621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8d74ace8609422dae2f2038ff8b5621" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 39841 } }
I20260928 06:33:02.688519 12820 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8d74ace8609422dae2f2038ff8b5621 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f8d74ace8609422dae2f2038ff8b5621" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8d74ace8609422dae2f2038ff8b5621" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 39841 } } }
I20260928 06:33:02.688683 12820 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8d74ace8609422dae2f2038ff8b5621 [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:02.688875 12820 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8d74ace8609422dae2f2038ff8b5621 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8d74ace8609422dae2f2038ff8b5621. Latest consensus state: current_term: 1 leader_uuid: "f8d74ace8609422dae2f2038ff8b5621" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8d74ace8609422dae2f2038ff8b5621" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 39841 } } }
I20260928 06:33:02.688938 12820 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8d74ace8609422dae2f2038ff8b5621 [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:02.688951 12828 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:33:02.689540 12828 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:33:02.691133 12828 catalog_manager.cc:1347] Generated new cluster ID: b7cfa1e99212450e898124d337008177
I20260928 06:33:02.691186 12828 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:33:02.691331 12824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:02.688191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47771 (local address 127.22.225.190:39841)
0928 06:33:02.688361 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:02.688372 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:02.689356 (+ 984us) server_negotiation.cc:407] Connection header received
0928 06:33:02.689590 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:02.689600 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:02.689633 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:02.689750 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:02.690825 (+ 1075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:02.690829 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:02.690829 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:33:02.690834 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:02.690875 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:33:02.690889 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:02.690941 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:02.691098 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 06:33:02.691145 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":76,"threads_started":1}
I20260928 06:33:02.692927 23430 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:33:02.699119 12824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:02.697965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33995 (local address 127.22.225.190:39841)
0928 06:33:02.698010 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:02.698012 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:02.698119 (+ 107us) server_negotiation.cc:407] Connection header received
0928 06:33:02.698218 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:02.698219 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:02.698224 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:02.698252 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:02.698879 (+ 627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:02.698881 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:02.698881 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:33:02.698885 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:02.698899 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:33:02.698903 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:02.698940 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:02.699039 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 06:33:02.699053 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 06:33:02.703519 12828 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:33:02.703701 12828 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:33:02.706596 12828 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f8d74ace8609422dae2f2038ff8b5621: Generated new TSK 0
I20260928 06:33:02.706776 12828 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:33:02.710186 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 12749
2026-09-28T06:33:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1421 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790576899143097-23430-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:33:02 dist-test-slave-d59m krb5kdc[12848](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:33:02 dist-test-slave-d59m krb5kdc[12848](info): set up 2 sockets
Sep 28 06:33:02 dist-test-slave-d59m krb5kdc[12848](info): commencing operation
krb5kdc: starting...
W20260928 06:33:03.773413 23430 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Sep 28 06:33:03 dist-test-slave-d59m krb5kdc[12848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:33:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:33:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab.
Entry for principal kudu/127.22.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab.
I20260928 06:33:03.897565 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790576899143097-23430-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790576899143097-23430-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790576899143097-23430-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:41077
--webserver_interface=127.22.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab
--principal=kudu/127.22.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:39899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.225.190:41077
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:03.980454 12861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:03.980767 12861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:03.980809 12861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:03.982075 12861 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:33:03.982136 12861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:03.982162 12861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:03.982183 12861 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:33:03.982223 12861 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:33:03.983726 12861 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:39899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790576899143097-23430-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790576899143097-23430-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.225.190:41077
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab
--principal=kudu/127.22.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:41077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12861
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790576899143097-23430-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:03.984000 12861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:03.984244 12861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:03.989066 12866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:03.989077 12869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:03.989144 12867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:03.989444 12861 server_base.cc:1034] running on GCE node
I20260928 06:33:03.989708 12861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:03.990037 12861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:03.991205 12861 hybrid_clock.cc:648] HybridClock initialized: now 1790577183991183 us; error 37 us; skew 500 ppm
Sep 28 06:33:03 dist-test-slave-d59m krb5kdc[12848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577183, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:03.993834 12861 init.cc:377] Logged in from keytab as kudu/127.22.225.190@KRBTEST.COM (short username kudu)
I20260928 06:33:03.994311 12861 webserver.cc:460] Webserver started at http://127.22.225.190:42373/ using document root <none> and password file <none>
I20260928 06:33:03.994561 12861 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:03.994632 12861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:03.994777 12861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:03.995812 12861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790576899143097-23430-0/minicluster-data/master-0/data/instance:
uuid: "f95359d3cd324807839f5876b7c6be12"
format_stamp: "Formatted at 2026-09-28 06:33:03 on dist-test-slave-d59m"
server_key: "db9fc1953d420631d81c0305997119ee"
server_key_iv: "b5cae86fc65e97f51a5c0fa085a312e8"
server_key_version: "encryptionkey@0"
I20260928 06:33:03.996206 12861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790576899143097-23430-0/minicluster-data/master-0/wal/instance:
uuid: "f95359d3cd324807839f5876b7c6be12"
format_stamp: "Formatted at 2026-09-28 06:33:03 on dist-test-slave-d59m"
server_key: "db9fc1953d420631d81c0305997119ee"
server_key_iv: "b5cae86fc65e97f51a5c0fa085a312e8"
server_key_version: "encryptionkey@0"
I20260928 06:33:03.997709 12861 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:33:03.998535 12876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:03.998710 12861 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:33:03.998816 12861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790576899143097-23430-0/minicluster-data/master-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790576899143097-23430-0/minicluster-data/master-0/wal
uuid: "f95359d3cd324807839f5876b7c6be12"
format_stamp: "Formatted at 2026-09-28 06:33:03 on dist-test-slave-d59m"
server_key: "db9fc1953d420631d81c0305997119ee"
server_key_iv: "b5cae86fc65e97f51a5c0fa085a312e8"
server_key_version: "encryptionkey@0"
I20260928 06:33:03.998891 12861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790576899143097-23430-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790576899143097-23430-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790576899143097-23430-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:04.014935 12861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:04.017184 12861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:04.017357 12861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:04.021767 12861 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.190:41077
I20260928 06:33:04.021802 12928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.190:41077 every 8 connection(s)
I20260928 06:33:04.022169 12861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
I20260928 06:33:04.022898 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 12861
I20260928 06:33:04.023008 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790576899143097-23430-0/minicluster-data/master-0/wal/instance
I20260928 06:33:04.023219 23430 external_mini_cluster.cc:1428] Setting key f1b5ebbf17682c1bf236292fb35b33c4
I20260928 06:33:04.023288 12929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:04.026813 12929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f95359d3cd324807839f5876b7c6be12: Bootstrap starting.
I20260928 06:33:04.027480 12929 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f95359d3cd324807839f5876b7c6be12: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:04.027769 12929 log.cc:824] T 00000000000000000000000000000000 P f95359d3cd324807839f5876b7c6be12: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:04.028477 12929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f95359d3cd324807839f5876b7c6be12: No bootstrap required, opened a new log
Sep 28 06:33:04 dist-test-slave-d59m krb5kdc[12848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:33:04.031131 12929 raft_consensus.cc:348] T 00000000000000000000000000000000 P f95359d3cd324807839f5876b7c6be12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f95359d3cd324807839f5876b7c6be12" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 41077 } }
I20260928 06:33:04.031315 12929 raft_consensus.cc:374] T 00000000000000000000000000000000 P f95359d3cd324807839f5876b7c6be12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:04.031394 12929 raft_consensus.cc:729] T 00000000000000000000000000000000 P f95359d3cd324807839f5876b7c6be12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f95359d3cd324807839f5876b7c6be12, State: Initialized, Role: FOLLOWER
I20260928 06:33:04.031541 12929 consensus_queue.cc:260] T 00000000000000000000000000000000 P f95359d3cd324807839f5876b7c6be12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f95359d3cd324807839f5876b7c6be12" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 41077 } }
I20260928 06:33:04.031642 12929 raft_consensus.cc:388] T 00000000000000000000000000000000 P f95359d3cd324807839f5876b7c6be12 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:33:04.031718 12929 raft_consensus.cc:482] T 00000000000000000000000000000000 P f95359d3cd324807839f5876b7c6be12 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:33:04.031771 12929 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f95359d3cd324807839f5876b7c6be12 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:04.032459 12929 raft_consensus.cc:504] T 00000000000000000000000000000000 P f95359d3cd324807839f5876b7c6be12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f95359d3cd324807839f5876b7c6be12" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 41077 } }
I20260928 06:33:04.032618 12929 leader_election.cc:302] T 00000000000000000000000000000000 P f95359d3cd324807839f5876b7c6be12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f95359d3cd324807839f5876b7c6be12; no voters:
I20260928 06:33:04.032804 12929 leader_election.cc:288] T 00000000000000000000000000000000 P f95359d3cd324807839f5876b7c6be12 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:33:04.032835 12934 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f95359d3cd324807839f5876b7c6be12 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:33:04.033041 12934 raft_consensus.cc:686] T 00000000000000000000000000000000 P f95359d3cd324807839f5876b7c6be12 [term 1 LEADER]: Becoming Leader. State: Replica: f95359d3cd324807839f5876b7c6be12, State: Running, Role: LEADER
I20260928 06:33:04.033206 12929 sys_catalog.cc:564] T 00000000000000000000000000000000 P f95359d3cd324807839f5876b7c6be12 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:33:04.033185 12934 consensus_queue.cc:237] T 00000000000000000000000000000000 P f95359d3cd324807839f5876b7c6be12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f95359d3cd324807839f5876b7c6be12" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 41077 } }
I20260928 06:33:04.035044 12935 sys_catalog.cc:455] T 00000000000000000000000000000000 P f95359d3cd324807839f5876b7c6be12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f95359d3cd324807839f5876b7c6be12. Latest consensus state: current_term: 1 leader_uuid: "f95359d3cd324807839f5876b7c6be12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f95359d3cd324807839f5876b7c6be12" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 41077 } } }
I20260928 06:33:04.035132 12935 sys_catalog.cc:458] T 00000000000000000000000000000000 P f95359d3cd324807839f5876b7c6be12 [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:04.035504 12936 sys_catalog.cc:455] T 00000000000000000000000000000000 P f95359d3cd324807839f5876b7c6be12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f95359d3cd324807839f5876b7c6be12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f95359d3cd324807839f5876b7c6be12" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 41077 } } }
I20260928 06:33:04.035578 12936 sys_catalog.cc:458] T 00000000000000000000000000000000 P f95359d3cd324807839f5876b7c6be12 [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:04.035601 12948 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:33:04.035840 12932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:04.023788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58365 (local address 127.22.225.190:41077)
0928 06:33:04.024033 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:04.024043 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:04.024076 (+ 33us) server_negotiation.cc:407] Connection header received
0928 06:33:04.024665 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:04.024674 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:04.024748 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:04.024910 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:04.025653 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:04.027020 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:04.028036 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:04.028619 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:04.028724 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:04.031609 (+ 2885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:04.031614 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:04.031614 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:04.031626 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:04.033672 (+ 2046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:04.034069 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:04.034072 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:04.034072 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:04.034172 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:04.034367 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:04.034367 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:04.034368 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:04.034657 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:04.035226 (+ 569us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:04.035296 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 06:33:04.035427 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":87,"threads_started":1}
I20260928 06:33:04.036080 12948 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:33:04.037002 23430 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:33:04.038757 12948 catalog_manager.cc:1347] Generated new cluster ID: 1ef3175db79f4780b22eeffe583b3613
I20260928 06:33:04.038837 12948 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:33:04.045644 12948 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:33:04.046317 12948 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:33:04.053100 12948 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f95359d3cd324807839f5876b7c6be12: Generated new TSK 0
I20260928 06:33:04.053334 12948 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:33:04.053555 12932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:04.044948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39703 (local address 127.22.225.190:41077)
0928 06:33:04.044990 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:04.044992 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:04.045531 (+ 539us) server_negotiation.cc:407] Connection header received
0928 06:33:04.045639 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:04.045640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:04.045650 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:04.045681 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:04.046595 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:04.047339 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:04.048408 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:04.048903 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:04.048948 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:04.050533 (+ 1585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:04.050534 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:04.050535 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:04.050542 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:04.052233 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:04.052678 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:04.052679 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:04.052679 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:04.052736 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:04.052940 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:04.052941 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:04.052942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:04.053065 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:04.053137 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:04.053347 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 06:33:04.053439 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 06:33:04.055877 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 12861
2026-09-28T06:33:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1345 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790576899143097-23430-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:33:04 dist-test-slave-d59m krb5kdc[12961](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:33:04 dist-test-slave-d59m krb5kdc[12961](info): set up 2 sockets
Sep 28 06:33:04 dist-test-slave-d59m krb5kdc[12961](info): commencing operation
krb5kdc: starting...
W20260928 06:33:05.101202 23430 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Sep 28 06:33:05 dist-test-slave-d59m krb5kdc[12961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:33:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:33:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab.
Entry for principal kudu/127.22.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab.
I20260928 06:33:05.223105 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790576899143097-23430-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790576899143097-23430-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790576899143097-23430-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:36161
--webserver_interface=127.22.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab
--principal=kudu/127.22.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:35861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.225.190:36161
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:05.305006 12974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:05.305239 12974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:05.305294 12974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:05.306464 12974 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:33:05.306532 12974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:05.306555 12974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:05.306586 12974 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:33:05.306634 12974 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:33:05.308138 12974 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:35861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790576899143097-23430-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790576899143097-23430-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.225.190:36161
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab
--principal=kudu/127.22.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:36161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790576899143097-23430-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:05.308423 12974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:05.308692 12974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:05.313659 12980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:05.313666 12982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:05.313879 12974 server_base.cc:1034] running on GCE node
W20260928 06:33:05.313653 12979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:05.314224 12974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:05.314580 12974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:05.315742 12974 hybrid_clock.cc:648] HybridClock initialized: now 1790577185315731 us; error 27 us; skew 500 ppm
Sep 28 06:33:05 dist-test-slave-d59m krb5kdc[12961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577185, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:05.318292 12974 init.cc:377] Logged in from keytab as kudu/127.22.225.190@KRBTEST.COM (short username kudu)
I20260928 06:33:05.318810 12974 webserver.cc:460] Webserver started at http://127.22.225.190:38011/ using document root <none> and password file <none>
I20260928 06:33:05.319052 12974 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:05.319098 12974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:05.319244 12974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:05.320327 12974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790576899143097-23430-0/minicluster-data/master-0/data/instance:
uuid: "bae660e308d64f669370cebdcad20b02"
format_stamp: "Formatted at 2026-09-28 06:33:05 on dist-test-slave-d59m"
server_key: "2ae0834b40ef3979f0a617306fb90b11"
server_key_iv: "cae1610f26aba4e828f4d40307a40b1c"
server_key_version: "encryptionkey@0"
I20260928 06:33:05.320721 12974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790576899143097-23430-0/minicluster-data/master-0/wal/instance:
uuid: "bae660e308d64f669370cebdcad20b02"
format_stamp: "Formatted at 2026-09-28 06:33:05 on dist-test-slave-d59m"
server_key: "2ae0834b40ef3979f0a617306fb90b11"
server_key_iv: "cae1610f26aba4e828f4d40307a40b1c"
server_key_version: "encryptionkey@0"
I20260928 06:33:05.322081 12974 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:33:05.322855 12989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:05.323001 12974 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:33:05.323091 12974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790576899143097-23430-0/minicluster-data/master-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790576899143097-23430-0/minicluster-data/master-0/wal
uuid: "bae660e308d64f669370cebdcad20b02"
format_stamp: "Formatted at 2026-09-28 06:33:05 on dist-test-slave-d59m"
server_key: "2ae0834b40ef3979f0a617306fb90b11"
server_key_iv: "cae1610f26aba4e828f4d40307a40b1c"
server_key_version: "encryptionkey@0"
I20260928 06:33:05.323160 12974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790576899143097-23430-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790576899143097-23430-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790576899143097-23430-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:05.341413 12974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:05.343640 12974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:05.343796 12974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:05.348800 12974 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.190:36161
I20260928 06:33:05.348830 13041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.190:36161 every 8 connection(s)
I20260928 06:33:05.349225 12974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
I20260928 06:33:05.350164 13042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:05.352540 13042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bae660e308d64f669370cebdcad20b02: Bootstrap starting.
I20260928 06:33:05.353211 13042 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bae660e308d64f669370cebdcad20b02: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:05.353511 13042 log.cc:824] T 00000000000000000000000000000000 P bae660e308d64f669370cebdcad20b02: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:05.354316 13042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bae660e308d64f669370cebdcad20b02: No bootstrap required, opened a new log
I20260928 06:33:05.355947 13042 raft_consensus.cc:348] T 00000000000000000000000000000000 P bae660e308d64f669370cebdcad20b02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bae660e308d64f669370cebdcad20b02" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36161 } }
I20260928 06:33:05.356102 13042 raft_consensus.cc:374] T 00000000000000000000000000000000 P bae660e308d64f669370cebdcad20b02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:05.356127 13042 raft_consensus.cc:729] T 00000000000000000000000000000000 P bae660e308d64f669370cebdcad20b02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bae660e308d64f669370cebdcad20b02, State: Initialized, Role: FOLLOWER
I20260928 06:33:05.356297 13042 consensus_queue.cc:260] T 00000000000000000000000000000000 P bae660e308d64f669370cebdcad20b02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bae660e308d64f669370cebdcad20b02" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36161 } }
I20260928 06:33:05.356364 13042 raft_consensus.cc:388] T 00000000000000000000000000000000 P bae660e308d64f669370cebdcad20b02 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:33:05.356420 13042 raft_consensus.cc:482] T 00000000000000000000000000000000 P bae660e308d64f669370cebdcad20b02 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:33:05.356477 13042 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bae660e308d64f669370cebdcad20b02 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:05.357424 13042 raft_consensus.cc:504] T 00000000000000000000000000000000 P bae660e308d64f669370cebdcad20b02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bae660e308d64f669370cebdcad20b02" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36161 } }
I20260928 06:33:05.357513 13042 leader_election.cc:302] T 00000000000000000000000000000000 P bae660e308d64f669370cebdcad20b02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bae660e308d64f669370cebdcad20b02; no voters:
I20260928 06:33:05.357709 13042 leader_election.cc:288] T 00000000000000000000000000000000 P bae660e308d64f669370cebdcad20b02 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:33:05.357767 13045 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bae660e308d64f669370cebdcad20b02 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:33:05.357928 13045 raft_consensus.cc:686] T 00000000000000000000000000000000 P bae660e308d64f669370cebdcad20b02 [term 1 LEADER]: Becoming Leader. State: Replica: bae660e308d64f669370cebdcad20b02, State: Running, Role: LEADER
I20260928 06:33:05.357990 13042 sys_catalog.cc:564] T 00000000000000000000000000000000 P bae660e308d64f669370cebdcad20b02 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:33:05.358331 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 12974
I20260928 06:33:05.358450 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790576899143097-23430-0/minicluster-data/master-0/wal/instance
I20260928 06:33:05.358641 23430 external_mini_cluster.cc:1428] Setting key 00caa9616ac51353da8c3d1a4593213b
I20260928 06:33:05.358085 13045 consensus_queue.cc:237] T 00000000000000000000000000000000 P bae660e308d64f669370cebdcad20b02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bae660e308d64f669370cebdcad20b02" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36161 } }
I20260928 06:33:05.359417 13046 sys_catalog.cc:455] T 00000000000000000000000000000000 P bae660e308d64f669370cebdcad20b02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bae660e308d64f669370cebdcad20b02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bae660e308d64f669370cebdcad20b02" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36161 } } }
I20260928 06:33:05.359506 13046 sys_catalog.cc:458] T 00000000000000000000000000000000 P bae660e308d64f669370cebdcad20b02 [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:05.359774 13047 sys_catalog.cc:455] T 00000000000000000000000000000000 P bae660e308d64f669370cebdcad20b02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bae660e308d64f669370cebdcad20b02. Latest consensus state: current_term: 1 leader_uuid: "bae660e308d64f669370cebdcad20b02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bae660e308d64f669370cebdcad20b02" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36161 } } }
I20260928 06:33:05.359915 13047 sys_catalog.cc:458] T 00000000000000000000000000000000 P bae660e308d64f669370cebdcad20b02 [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:05.360092 13056 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:33:05.361014 13056 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:33:05.363545 13056 catalog_manager.cc:1347] Generated new cluster ID: 92f813790b734125a28ce25fe4d607c3
I20260928 06:33:05.363598 13056 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:33:05.373423 13056 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:33:05.374243 13056 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:33:05.374249 13055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:05.359308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48209 (local address 127.22.225.190:36161)
0928 06:33:05.359517 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:05.359527 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:05.359770 (+ 243us) server_negotiation.cc:407] Connection header received
0928 06:33:05.359897 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:05.359905 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:05.359963 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:05.360107 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:05.360779 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:05.362070 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:05.362921 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:05.363520 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:05.363964 (+ 444us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:05.373222 (+ 9258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:05.373227 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:05.373227 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:33:05.373478 (+ 251us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:05.373527 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:33:05.373546 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:05.373897 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:05.373931 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 06:33:05.374002 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":85,"threads_started":1}
I20260928 06:33:05.375273 23430 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:33:05.384755 13056 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bae660e308d64f669370cebdcad20b02: Generated new TSK 0
I20260928 06:33:05.385084 13056 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:33:05.388123 13055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:05.384748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34115 (local address 127.22.225.190:36161)
0928 06:33:05.384790 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:05.384792 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:05.385165 (+ 373us) server_negotiation.cc:407] Connection header received
0928 06:33:05.385217 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:05.385219 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:05.385232 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:05.385284 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:05.385498 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:05.386004 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:05.386797 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:05.387021 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:05.387114 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:05.387835 (+ 721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:05.387836 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:05.387837 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:33:05.387841 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:05.387857 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:33:05.387862 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:05.387931 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:05.387989 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 06:33:05.388016 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 06:33:05.390215 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 12974
2026-09-28T06:33:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1334 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790576899143097-23430-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:33:05 dist-test-slave-d59m krb5kdc[13073](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:33:05 dist-test-slave-d59m krb5kdc[13073](info): set up 2 sockets
Sep 28 06:33:05 dist-test-slave-d59m krb5kdc[13073](info): commencing operation
krb5kdc: starting...
W20260928 06:33:06.461366 23430 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Sep 28 06:33:06 dist-test-slave-d59m krb5kdc[13073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:33:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:33:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab.
Entry for principal kudu/127.22.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab.
I20260928 06:33:06.593592 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790576899143097-23430-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790576899143097-23430-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790576899143097-23430-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:34143
--webserver_interface=127.22.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab
--principal=kudu/127.22.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:33939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.225.190:34143
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:06.691854 13086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:06.692065 13086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:06.692090 13086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:06.693666 13086 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:33:06.693715 13086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:06.693732 13086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:06.693763 13086 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:33:06.693789 13086 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:33:06.695664 13086 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:33939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790576899143097-23430-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790576899143097-23430-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.225.190:34143
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab
--principal=kudu/127.22.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:34143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790576899143097-23430-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:06.695957 13086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:06.696161 13086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:06.701421 13091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:06.701452 13092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:06.701601 13094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:06.701777 13086 server_base.cc:1034] running on GCE node
I20260928 06:33:06.702104 13086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:06.702482 13086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:06.703666 13086 hybrid_clock.cc:648] HybridClock initialized: now 1790577186703638 us; error 49 us; skew 500 ppm
Sep 28 06:33:06 dist-test-slave-d59m krb5kdc[13073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577186, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:06.711805 13086 init.cc:377] Logged in from keytab as kudu/127.22.225.190@KRBTEST.COM (short username kudu)
I20260928 06:33:06.712324 13086 webserver.cc:460] Webserver started at http://127.22.225.190:44577/ using document root <none> and password file <none>
I20260928 06:33:06.712639 13086 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:06.712711 13086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:06.712858 13086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:06.713933 13086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790576899143097-23430-0/minicluster-data/master-0/data/instance:
uuid: "1512ff12cf174ec6bd6e1104ebdb5d70"
format_stamp: "Formatted at 2026-09-28 06:33:06 on dist-test-slave-d59m"
server_key: "a1c180b56335b4d51ee1bf06828bf1e5"
server_key_iv: "45be88d0ec9497649d424151f47c611e"
server_key_version: "encryptionkey@0"
I20260928 06:33:06.714340 13086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790576899143097-23430-0/minicluster-data/master-0/wal/instance:
uuid: "1512ff12cf174ec6bd6e1104ebdb5d70"
format_stamp: "Formatted at 2026-09-28 06:33:06 on dist-test-slave-d59m"
server_key: "a1c180b56335b4d51ee1bf06828bf1e5"
server_key_iv: "45be88d0ec9497649d424151f47c611e"
server_key_version: "encryptionkey@0"
I20260928 06:33:06.715934 13086 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:33:06.716817 13101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:06.716995 13086 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:33:06.717093 13086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790576899143097-23430-0/minicluster-data/master-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790576899143097-23430-0/minicluster-data/master-0/wal
uuid: "1512ff12cf174ec6bd6e1104ebdb5d70"
format_stamp: "Formatted at 2026-09-28 06:33:06 on dist-test-slave-d59m"
server_key: "a1c180b56335b4d51ee1bf06828bf1e5"
server_key_iv: "45be88d0ec9497649d424151f47c611e"
server_key_version: "encryptionkey@0"
I20260928 06:33:06.717185 13086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790576899143097-23430-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790576899143097-23430-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790576899143097-23430-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:06.721037 13086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:06.723456 13086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:06.723600 13086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:06.728734 13086 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.190:34143
I20260928 06:33:06.728778 13153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.190:34143 every 8 connection(s)
I20260928 06:33:06.729151 13086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
I20260928 06:33:06.730296 13154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:06.733510 13154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1512ff12cf174ec6bd6e1104ebdb5d70: Bootstrap starting.
I20260928 06:33:06.734120 13154 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1512ff12cf174ec6bd6e1104ebdb5d70: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:06.734416 13154 log.cc:824] T 00000000000000000000000000000000 P 1512ff12cf174ec6bd6e1104ebdb5d70: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:06.735296 13154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1512ff12cf174ec6bd6e1104ebdb5d70: No bootstrap required, opened a new log
I20260928 06:33:06.738154 13154 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1512ff12cf174ec6bd6e1104ebdb5d70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1512ff12cf174ec6bd6e1104ebdb5d70" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 34143 } }
I20260928 06:33:06.738390 13154 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1512ff12cf174ec6bd6e1104ebdb5d70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:06.738444 13154 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1512ff12cf174ec6bd6e1104ebdb5d70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1512ff12cf174ec6bd6e1104ebdb5d70, State: Initialized, Role: FOLLOWER
I20260928 06:33:06.738598 13154 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1512ff12cf174ec6bd6e1104ebdb5d70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1512ff12cf174ec6bd6e1104ebdb5d70" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 34143 } }
I20260928 06:33:06.738692 13154 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1512ff12cf174ec6bd6e1104ebdb5d70 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:33:06.738754 13154 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1512ff12cf174ec6bd6e1104ebdb5d70 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:33:06.738818 13154 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1512ff12cf174ec6bd6e1104ebdb5d70 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:06.739290 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 13086
I20260928 06:33:06.739423 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790576899143097-23430-0/minicluster-data/master-0/wal/instance
I20260928 06:33:06.739679 23430 external_mini_cluster.cc:1428] Setting key 8bebaa9f491f9eff34cb952ca8a1dbcf
I20260928 06:33:06.740031 13154 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1512ff12cf174ec6bd6e1104ebdb5d70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1512ff12cf174ec6bd6e1104ebdb5d70" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 34143 } }
I20260928 06:33:06.740222 13154 leader_election.cc:302] T 00000000000000000000000000000000 P 1512ff12cf174ec6bd6e1104ebdb5d70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1512ff12cf174ec6bd6e1104ebdb5d70; no voters:
I20260928 06:33:06.740460 13159 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1512ff12cf174ec6bd6e1104ebdb5d70 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:33:06.740458 13154 leader_election.cc:288] T 00000000000000000000000000000000 P 1512ff12cf174ec6bd6e1104ebdb5d70 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:33:06.740666 13159 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1512ff12cf174ec6bd6e1104ebdb5d70 [term 1 LEADER]: Becoming Leader. State: Replica: 1512ff12cf174ec6bd6e1104ebdb5d70, State: Running, Role: LEADER
I20260928 06:33:06.740803 13159 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1512ff12cf174ec6bd6e1104ebdb5d70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1512ff12cf174ec6bd6e1104ebdb5d70" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 34143 } }
I20260928 06:33:06.740895 13154 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1512ff12cf174ec6bd6e1104ebdb5d70 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:33:06.741236 13159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1512ff12cf174ec6bd6e1104ebdb5d70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1512ff12cf174ec6bd6e1104ebdb5d70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1512ff12cf174ec6bd6e1104ebdb5d70" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 34143 } } }
I20260928 06:33:06.741362 13159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1512ff12cf174ec6bd6e1104ebdb5d70 [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:06.741246 13160 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1512ff12cf174ec6bd6e1104ebdb5d70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1512ff12cf174ec6bd6e1104ebdb5d70. Latest consensus state: current_term: 1 leader_uuid: "1512ff12cf174ec6bd6e1104ebdb5d70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1512ff12cf174ec6bd6e1104ebdb5d70" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 34143 } } }
I20260928 06:33:06.741606 13160 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1512ff12cf174ec6bd6e1104ebdb5d70 [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:06.741984 13166 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:33:06.742744 13166 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:33:06.743101 13158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:06.740212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38079 (local address 127.22.225.190:34143)
0928 06:33:06.740436 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:06.740447 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:06.740486 (+ 39us) server_negotiation.cc:407] Connection header received
0928 06:33:06.740668 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:06.740676 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:06.740711 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:06.740862 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:06.742527 (+ 1665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:06.742531 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:06.742531 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:33:06.742542 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:06.742591 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:33:06.742603 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:06.742681 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:06.742902 (+ 221us) server_negotiation.cc:299] Negotiation successful
0928 06:33:06.742945 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":104,"threads_started":1}
I20260928 06:33:06.744220 23430 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:33:06.746413 13166 catalog_manager.cc:1347] Generated new cluster ID: a6739bf23c8847979f147702ac669fe4
I20260928 06:33:06.746486 13166 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:33:06.752290 13166 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:33:06.752457 13166 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:33:06.761322 13166 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1512ff12cf174ec6bd6e1104ebdb5d70: Generated new TSK 0
I20260928 06:33:06.761490 13158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:06.745730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41315 (local address 127.22.225.190:34143)
0928 06:33:06.745774 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:06.745776 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:06.752785 (+ 7009us) server_negotiation.cc:407] Connection header received
0928 06:33:06.752896 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:06.752898 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:06.752904 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:06.752935 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:06.761222 (+ 8287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:06.761224 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:06.761225 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:33:06.761230 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:06.761250 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:33:06.761256 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:06.761288 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:06.761397 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 06:33:06.761412 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 06:33:06.761582 13166 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:33:06.764122 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 13086
2026-09-28T06:33:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1374 ms)
[----------] 18 tests from AuthTokenIssuingTest (25313 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:33:06 dist-test-slave-d59m krb5kdc[13185](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:33:06 dist-test-slave-d59m krb5kdc[13185](info): set up 2 sockets
Sep 28 06:33:06 dist-test-slave-d59m krb5kdc[13185](info): commencing operation
krb5kdc: starting...
W20260928 06:33:07.929419 23430 mini_kdc.cc:121] Time spent starting KDC: real 1.145s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Sep 28 06:33:08 dist-test-slave-d59m krb5kdc[13185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:33:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:33:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab.
Entry for principal kudu/127.22.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab.
I20260928 06:33:08.059487 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:36923
--webserver_interface=127.22.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab
--principal=kudu/127.22.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:38237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.225.190:36923,127.22.225.189:37297,127.22.225.188:35927
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:08.161188 13198 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 06:33:08.161425 13198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:08.161444 13198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:08.161468 13198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:08.162761 13198 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:33:08.162809 13198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:08.162824 13198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:08.162837 13198 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:33:08.162850 13198 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:33:08.164466 13198 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:38237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.225.190:36923,127.22.225.189:37297,127.22.225.188:35927
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab
--principal=kudu/127.22.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:36923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:08.164808 13198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:08.165069 13198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:08.170158 13203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:08.170154 13204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:08.170397 13198 server_base.cc:1034] running on GCE node
W20260928 06:33:08.170173 13206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:08.170753 13198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:08.171128 13198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:08.172307 13198 hybrid_clock.cc:648] HybridClock initialized: now 1790577188172292 us; error 33 us; skew 500 ppm
Sep 28 06:33:08 dist-test-slave-d59m krb5kdc[13185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577188, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:08.175272 13198 init.cc:377] Logged in from keytab as kudu/127.22.225.190@KRBTEST.COM (short username kudu)
I20260928 06:33:08.175834 13198 webserver.cc:460] Webserver started at http://127.22.225.190:39111/ using document root <none> and password file <none>
I20260928 06:33:08.176115 13198 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:08.176178 13198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:08.176303 13198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:08.177781 13198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/minicluster-data/master-0/data/instance:
uuid: "83509909ca404cc5b5edbd57811dd113"
format_stamp: "Formatted at 2026-09-28 06:33:08 on dist-test-slave-d59m"
server_key: "149faa1f51812996dd5e332ba905cbfa"
server_key_iv: "274730f690d1a34b199845a1b1670394"
server_key_version: "encryptionkey@0"
I20260928 06:33:08.178303 13198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/minicluster-data/master-0/wal/instance:
uuid: "83509909ca404cc5b5edbd57811dd113"
format_stamp: "Formatted at 2026-09-28 06:33:08 on dist-test-slave-d59m"
server_key: "149faa1f51812996dd5e332ba905cbfa"
server_key_iv: "274730f690d1a34b199845a1b1670394"
server_key_version: "encryptionkey@0"
I20260928 06:33:08.180140 13198 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.003s
I20260928 06:33:08.181206 13213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:08.181409 13198 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:33:08.181529 13198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/minicluster-data/master-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/minicluster-data/master-0/wal
uuid: "83509909ca404cc5b5edbd57811dd113"
format_stamp: "Formatted at 2026-09-28 06:33:08 on dist-test-slave-d59m"
server_key: "149faa1f51812996dd5e332ba905cbfa"
server_key_iv: "274730f690d1a34b199845a1b1670394"
server_key_version: "encryptionkey@0"
I20260928 06:33:08.181612 13198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:08.195763 13198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:08.198678 13198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:08.198801 13198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:08.203604 13198 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.190:36923
I20260928 06:33:08.203668 13265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.190:36923 every 8 connection(s)
I20260928 06:33:08.204037 13198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
I20260928 06:33:08.205240 13266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:08.205338 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 13198
I20260928 06:33:08.205461 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/minicluster-data/master-0/wal/instance
I20260928 06:33:08.205615 23430 external_mini_cluster.cc:1428] Setting key 3eb580357bab03bcf7741901832fe1d0
I20260928 06:33:08.207858 13266 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36923 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.22.225.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.189@KRBTEST.COM" created.
Sep 28 06:33:08 dist-test-slave-d59m krb5kdc[13185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577188, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.190@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:33:08.222776 13270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:08.208860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36923 (local address 127.0.0.1:56708)
0928 06:33:08.209176 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:08.209219 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:08.209941 (+ 722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:08.210361 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:08.210365 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:08.210866 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:08.211130 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:08.211138 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:08.212757 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:08.212758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:08.214076 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:08.214081 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:08.214613 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:08.214656 (+ 43us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:08.214758 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:08.215327 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:08.215340 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:08.217997 (+ 2657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:08.220626 (+ 2629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:08.220635 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:08.220638 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:08.220966 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:08.221292 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:08.221292 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:08.221293 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:08.221426 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:08.221955 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:08.221963 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:08.222057 (+ 94us) client_negotiation.cc:769] Sending connection context
0928 06:33:08.222179 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 06:33:08.222516 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"mutex_wait_us":125,"thread_start_us":102,"threads_started":1}
I20260928 06:33:08.222787 13269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:08.208878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56708 (local address 127.22.225.190:36923)
0928 06:33:08.209098 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:08.209109 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:08.210106 (+ 997us) server_negotiation.cc:407] Connection header received
0928 06:33:08.210192 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:08.210194 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:08.210220 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:08.210316 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:08.211264 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:08.212652 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:08.214193 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:08.214511 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:08.214596 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:08.218152 (+ 3556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:08.218161 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:08.218162 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:08.218171 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:08.220472 (+ 2301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:08.221077 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:08.221081 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:08.221082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:08.221207 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:08.221513 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:08.221514 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:08.221514 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:08.221864 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:08.221970 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:08.222236 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 06:33:08.222393 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":91,"threads_started":1}
I20260928 06:33:08.223933 13266 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.225.189" port: 37297 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:33:08.224363 13270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:08.224205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.189:37297 (local address 127.0.0.1:46140)
0928 06:33:08.224263 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:08.224324 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.189:37297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 06:33:08.224649 13216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.225.189:37297: connect: Connection refused (error 111)
I20260928 06:33:08.225168 13270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:08.225051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.189:37297 (local address 127.0.0.1:46142)
0928 06:33:08.225105 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:08.225138 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.189:37297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 06:33:08.225293 13266 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.225.189:37297: Network error: Client connection negotiation failed: client connection to 127.22.225.189:37297: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.189.keytab.
Entry for principal kudu/127.22.225.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.189.keytab.
I20260928 06:33:08.235172 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.189:37297
--webserver_interface=127.22.225.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.189.keytab
--principal=kudu/127.22.225.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:38237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.225.190:36923,127.22.225.189:37297,127.22.225.188:35927
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:33:08.274521 13266 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.225.189" port: 37297 } attempt: 1
I20260928 06:33:08.275090 13270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:08.274896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.189:37297 (local address 127.0.0.1:46158)
0928 06:33:08.274992 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:08.275039 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.189:37297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260928 06:33:08.275532 13270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:08.275339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.189:37297 (local address 127.0.0.1:46164)
0928 06:33:08.275477 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:08.275507 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.189:37297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
W20260928 06:33:08.275625 13266 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.225.189:37297: Network error: Client connection negotiation failed: client connection to 127.22.225.189:37297: connect: Connection refused (error 111)
W20260928 06:33:08.325621 13273 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 06:33:08.325888 13273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:08.325932 13273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:08.325973 13273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:08.327183 13273 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:33:08.327255 13273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:08.327282 13273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:08.327313 13273 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:33:08.327340 13273 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:33:08.329005 13273 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:38237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.22.225.190:36923,127.22.225.189:37297,127.22.225.188:35927
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.189.keytab
--principal=kudu/127.22.225.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.189:37297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.22.225.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13273
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:08.329407 13273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:08.329686 13273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:08.335947 13278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:08.336748 13273 server_base.cc:1034] running on GCE node
W20260928 06:33:08.336918 13281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:08.336966 13279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:08.337448 13273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:08.337839 13273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:08.339061 13273 hybrid_clock.cc:648] HybridClock initialized: now 1790577188339045 us; error 47 us; skew 500 ppm
Sep 28 06:33:08 dist-test-slave-d59m krb5kdc[13185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577188, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:08.342454 13273 init.cc:377] Logged in from keytab as kudu/127.22.225.189@KRBTEST.COM (short username kudu)
I20260928 06:33:08.343111 13273 webserver.cc:460] Webserver started at http://127.22.225.189:43613/ using document root <none> and password file <none>
I20260928 06:33:08.343451 13273 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:08.343561 13273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:08.343781 13266 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.225.189" port: 37297 } attempt: 2
I20260928 06:33:08.344175 13273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:08.344249 13270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:08.344096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.189:37297 (local address 127.0.0.1:46166)
0928 06:33:08.344163 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:08.344208 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.189:37297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 06:33:08.344620 13270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:08.344526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.189:37297 (local address 127.0.0.1:46172)
0928 06:33:08.344559 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:08.344597 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.189:37297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260928 06:33:08.344745 13266 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.225.189:37297: Network error: Client connection negotiation failed: client connection to 127.22.225.189:37297: connect: Connection refused (error 111)
I20260928 06:33:08.345814 13273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/minicluster-data/master-1/data/instance:
uuid: "d5f2a3245b594c5ca7b7fc017ce03df4"
format_stamp: "Formatted at 2026-09-28 06:33:08 on dist-test-slave-d59m"
server_key: "902a3f1f1aea231c1607f082603baeb0"
server_key_iv: "476a1a302df9e4270e7a54f463e1f18e"
server_key_version: "encryptionkey@0"
I20260928 06:33:08.346316 13273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/minicluster-data/master-1/wal/instance:
uuid: "d5f2a3245b594c5ca7b7fc017ce03df4"
format_stamp: "Formatted at 2026-09-28 06:33:08 on dist-test-slave-d59m"
server_key: "902a3f1f1aea231c1607f082603baeb0"
server_key_iv: "476a1a302df9e4270e7a54f463e1f18e"
server_key_version: "encryptionkey@0"
I20260928 06:33:08.348264 13273 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.001s
I20260928 06:33:08.349426 13288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:08.349655 13273 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:33:08.349793 13273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/minicluster-data/master-1/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/minicluster-data/master-1/wal
uuid: "d5f2a3245b594c5ca7b7fc017ce03df4"
format_stamp: "Formatted at 2026-09-28 06:33:08 on dist-test-slave-d59m"
server_key: "902a3f1f1aea231c1607f082603baeb0"
server_key_iv: "476a1a302df9e4270e7a54f463e1f18e"
server_key_version: "encryptionkey@0"
I20260928 06:33:08.349929 13273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:08.369712 13273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:08.372942 13273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:08.373117 13273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:08.378258 13273 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.189:37297
I20260928 06:33:08.378278 13340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.189:37297 every 8 connection(s)
I20260928 06:33:08.378871 13273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/minicluster-data/master-1/data/info.pb
I20260928 06:33:08.379797 13341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:08.381517 13341 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36923 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:33:08.381765 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 13273
I20260928 06:33:08.381896 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/minicluster-data/master-1/wal/instance
I20260928 06:33:08.382081 23430 external_mini_cluster.cc:1428] Setting key ba00153530c009363c2ddaa84a11849a
Sep 28 06:33:08 dist-test-slave-d59m krb5kdc[13185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577188, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.189@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.225.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.188@KRBTEST.COM" created.
I20260928 06:33:08.395051 13269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:08.382287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56710 (local address 127.22.225.190:36923)
0928 06:33:08.382347 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:08.382349 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:08.382608 (+ 259us) server_negotiation.cc:407] Connection header received
0928 06:33:08.383127 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:08.383129 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:08.383142 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:08.383209 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:08.384144 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:08.385038 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:08.386692 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:08.387072 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:08.387163 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:08.390709 (+ 3546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:08.390712 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:08.390712 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:08.390722 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:08.392466 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:08.393347 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:08.393349 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:08.393350 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:08.393414 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:08.393713 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:08.393714 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:08.393715 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:08.393873 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:08.393960 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:08.394777 (+ 817us) server_negotiation.cc:299] Negotiation successful
0928 06:33:08.394883 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 06:33:08.396283 13343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:08.382293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36923 (local address 127.0.0.1:56710)
0928 06:33:08.382457 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:08.382499 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:08.383014 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:08.383289 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:08.383292 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:08.383717 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:08.384013 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:08.384020 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:08.385143 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:08.385144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:08.386570 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:08.386584 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:08.387181 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:08.387183 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:08.387357 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:08.388039 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:08.388051 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:08.390532 (+ 2481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:08.392880 (+ 2348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:08.392889 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:08.392893 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:08.393246 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:08.393490 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:08.393491 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:08.393491 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:08.393643 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:08.394166 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:08.394174 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:08.394481 (+ 307us) client_negotiation.cc:769] Sending connection context
0928 06:33:08.395608 (+ 1127us) client_negotiation.cc:240] Negotiation successful
0928 06:33:08.395956 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":89,"threads_started":1}
I20260928 06:33:08.396920 13341 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.225.189" port: 37297 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:33:08 dist-test-slave-d59m krb5kdc[13185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577188, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.189@KRBTEST.COM for kudu/127.22.225.189@KRBTEST.COM
I20260928 06:33:08.405220 13343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:08.397268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.189:37297 (local address 127.0.0.1:46180)
0928 06:33:08.397317 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:08.397327 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:08.397366 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:08.397669 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:08.397670 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:08.397845 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:08.397914 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:08.397916 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:08.399140 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:08.399141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:08.399651 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:08.399653 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:08.400153 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:08.400154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:08.400219 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:08.400689 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:08.400695 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:08.401971 (+ 1276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:08.403822 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:08.403823 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:08.403824 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:08.404111 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:08.404359 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:08.404360 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:08.404360 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:08.404447 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:08.404851 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:08.404852 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:08.404892 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 06:33:08.404988 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 06:33:08.405099 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 06:33:08.405360 13346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:08.397345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46180 (local address 127.22.225.189:37297)
0928 06:33:08.397494 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:08.397499 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:08.397523 (+ 24us) server_negotiation.cc:407] Connection header received
0928 06:33:08.397544 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:08.397546 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:08.397561 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:08.397644 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:08.398024 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:08.399048 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:08.399738 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:08.400072 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:08.400158 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:08.402069 (+ 1911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:08.402077 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:08.402077 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:08.402084 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:08.403717 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:08.404216 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:08.404222 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:08.404222 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:08.404284 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:08.404516 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:08.404517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:08.404517 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:08.404761 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:08.404866 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:08.405045 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 06:33:08.405154 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":118,"thread_start_us":83,"threads_started":1}
I20260928 06:33:08.406500 13341 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.225.188" port: 35927 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:33:08.406927 13343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:08.406767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.188:35927 (local address 127.0.0.1:34748)
0928 06:33:08.406826 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:08.406900 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.188:35927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 06:33:08.407068 13289 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.225.188:35927: connect: Connection refused (error 111)
I20260928 06:33:08.407600 13343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:08.407507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.188:35927 (local address 127.0.0.1:34754)
0928 06:33:08.407538 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:08.407578 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.188:35927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260928 06:33:08.407773 13341 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.225.188:35927: Network error: Client connection negotiation failed: client connection to 127.22.225.188:35927: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.188.keytab.
Entry for principal kudu/127.22.225.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.188.keytab.
I20260928 06:33:08.416159 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.188:35927
--webserver_interface=127.22.225.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.188.keytab
--principal=kudu/127.22.225.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:38237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.225.190:36923,127.22.225.189:37297,127.22.225.188:35927
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:33:08.435895 13266 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.225.189" port: 37297 } attempt: 3
Sep 28 06:33:08 dist-test-slave-d59m krb5kdc[13185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577188, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.190@KRBTEST.COM for kudu/127.22.225.189@KRBTEST.COM
I20260928 06:33:08.444471 13270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:08.436341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.189:37297 (local address 127.0.0.1:46196)
0928 06:33:08.436406 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:08.436417 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:08.436461 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:08.436669 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:08.436670 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:08.436862 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:08.436947 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:08.436949 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:08.437901 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:08.437902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:08.438739 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:08.438741 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:08.439280 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:08.439281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:08.439358 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:08.439871 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:08.439877 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:08.441448 (+ 1571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:08.443338 (+ 1890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:08.443339 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:08.443339 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:08.443569 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:08.443788 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:08.443789 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:08.443789 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:08.443856 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:08.444139 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:08.444140 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:08.444192 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 06:33:08.444265 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 06:33:08.444364 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 06:33:08.444471 13346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:08.436408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46196 (local address 127.22.225.189:37297)
0928 06:33:08.436467 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:08.436469 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:08.436482 (+ 13us) server_negotiation.cc:407] Connection header received
0928 06:33:08.436513 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:08.436514 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:08.436526 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:08.436658 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:08.437033 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:08.437805 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:08.438839 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:08.439194 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:08.439271 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:08.441548 (+ 2277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:08.441551 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:08.441551 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:08.441560 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:08.443246 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:08.443648 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:08.443649 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:08.443650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:08.443715 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:08.443924 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:08.443925 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:08.443926 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:08.444061 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:08.444136 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:08.444302 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 06:33:08.444382 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 06:33:08.445576 13266 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.225.188" port: 35927 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:33:08.446147 13270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:08.445924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.188:35927 (local address 127.0.0.1:34764)
0928 06:33:08.446058 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:08.446091 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.188:35927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260928 06:33:08.446537 13270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:08.446453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.188:35927 (local address 127.0.0.1:34776)
0928 06:33:08.446488 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:08.446511 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.188:35927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260928 06:33:08.446707 13266 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.225.188:35927: Network error: Client connection negotiation failed: client connection to 127.22.225.188:35927: connect: Connection refused (error 111)
I20260928 06:33:08.457562 13341 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.225.188" port: 35927 } attempt: 1
I20260928 06:33:08.458101 13343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:08.457927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.188:35927 (local address 127.0.0.1:34792)
0928 06:33:08.457994 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:08.458053 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.188:35927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 06:33:08.458503 13343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:08.458402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.188:35927 (local address 127.0.0.1:34794)
0928 06:33:08.458449 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:08.458477 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.188:35927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 06:33:08.458629 13341 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.225.188:35927: Network error: Client connection negotiation failed: client connection to 127.22.225.188:35927: connect: Connection refused (error 111)
I20260928 06:33:08.477880 13266 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.225.188" port: 35927 } attempt: 1
I20260928 06:33:08.478447 13270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:08.478249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.188:35927 (local address 127.0.0.1:34806)
0928 06:33:08.478339 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:08.478396 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.188:35927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 06:33:08.478842 13270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:08.478719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.188:35927 (local address 127.0.0.1:34822)
0928 06:33:08.478776 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:08.478810 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.188:35927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 06:33:08.478948 13266 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.225.188:35927: Network error: Client connection negotiation failed: client connection to 127.22.225.188:35927: connect: Connection refused (error 111)
W20260928 06:33:08.514546 13348 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 06:33:08.514882 13348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:08.514928 13348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:08.514966 13348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:08.517248 13348 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:33:08.517333 13348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:08.517354 13348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:08.517398 13348 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:33:08.517431 13348 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:33:08.520366 13348 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:38237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.22.225.190:36923,127.22.225.189:37297,127.22.225.188:35927
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.188.keytab
--principal=kudu/127.22.225.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.188:35927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.22.225.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13348
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:08.520741 13348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:08.521057 13348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:33:08.526769 13341 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.225.188" port: 35927 } attempt: 2
I20260928 06:33:08.527276 13343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:08.527120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.188:35927 (local address 127.0.0.1:34834)
0928 06:33:08.527179 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:08.527217 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.188:35927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260928 06:33:08.527634 13343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:08.527533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.188:35927 (local address 127.0.0.1:34850)
0928 06:33:08.527573 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:08.527609 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.188:35927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 06:33:08.527732 13341 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.225.188:35927: Network error: Client connection negotiation failed: client connection to 127.22.225.188:35927: connect: Connection refused (error 111)
W20260928 06:33:08.528858 13356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:08.528903 13354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:08.530164 13348 server_base.cc:1034] running on GCE node
W20260928 06:33:08.530742 13353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:08.531128 13348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:08.531713 13348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:08.533530 13348 hybrid_clock.cc:648] HybridClock initialized: now 1790577188533482 us; error 61 us; skew 500 ppm
Sep 28 06:33:08 dist-test-slave-d59m krb5kdc[13185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577188, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:08.539743 13348 init.cc:377] Logged in from keytab as kudu/127.22.225.188@KRBTEST.COM (short username kudu)
I20260928 06:33:08.540418 13348 webserver.cc:460] Webserver started at http://127.22.225.188:46035/ using document root <none> and password file <none>
I20260928 06:33:08.540855 13348 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:08.540927 13348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:08.541215 13348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:08.542994 13348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/minicluster-data/master-2/data/instance:
uuid: "6bcb789fca4e421f8da9f41b31334609"
format_stamp: "Formatted at 2026-09-28 06:33:08 on dist-test-slave-d59m"
server_key: "01dc1b53b49af892a98daddb7a398a58"
server_key_iv: "5f888f9cc3e702c15417028629d0ea00"
server_key_version: "encryptionkey@0"
I20260928 06:33:08.543466 13348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/minicluster-data/master-2/wal/instance:
uuid: "6bcb789fca4e421f8da9f41b31334609"
format_stamp: "Formatted at 2026-09-28 06:33:08 on dist-test-slave-d59m"
server_key: "01dc1b53b49af892a98daddb7a398a58"
server_key_iv: "5f888f9cc3e702c15417028629d0ea00"
server_key_version: "encryptionkey@0"
I20260928 06:33:08.546141 13348 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:33:08.547307 13363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:08.547524 13348 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:08.547650 13348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/minicluster-data/master-2/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/minicluster-data/master-2/wal
uuid: "6bcb789fca4e421f8da9f41b31334609"
format_stamp: "Formatted at 2026-09-28 06:33:08 on dist-test-slave-d59m"
server_key: "01dc1b53b49af892a98daddb7a398a58"
server_key_iv: "5f888f9cc3e702c15417028629d0ea00"
server_key_version: "encryptionkey@0"
I20260928 06:33:08.547760 13348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:08.554138 13266 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.225.188" port: 35927 } attempt: 2
I20260928 06:33:08.554600 13270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:08.554430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.188:35927 (local address 127.0.0.1:34864)
0928 06:33:08.554505 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:08.554551 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.188:35927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 06:33:08.555050 13270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:08.554933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.188:35927 (local address 127.0.0.1:34868)
0928 06:33:08.554986 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:08.555018 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.188:35927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 06:33:08.555186 13266 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.225.188:35927: Network error: Client connection negotiation failed: client connection to 127.22.225.188:35927: connect: Connection refused (error 111)
I20260928 06:33:08.564503 13348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:08.567533 13348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:08.567765 13348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:08.575131 13348 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.188:35927
I20260928 06:33:08.575161 13415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.188:35927 every 8 connection(s)
I20260928 06:33:08.575681 13348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/minicluster-data/master-2/data/info.pb
I20260928 06:33:08.576948 13416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:08.578547 13416 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36923 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:33:08.582783 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 13348
I20260928 06:33:08.582897 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576899143097-23430-0/minicluster-data/master-2/wal/instance
I20260928 06:33:08.583091 23430 external_mini_cluster.cc:1428] Setting key 2bf631799eb0d2b883a787f15013a072
Sep 28 06:33:08 dist-test-slave-d59m krb5kdc[13185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577188, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.188@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
Sep 28 06:33:08 dist-test-slave-d59m krb5kdc[13185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.225.188@KRBTEST.COM
I20260928 06:33:08.592381 13269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:08.579701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56722 (local address 127.22.225.190:36923)
0928 06:33:08.579748 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:08.579751 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:08.579762 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:33:08.580423 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:08.580425 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:08.580441 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:08.580515 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:08.581605 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:08.582449 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:08.584308 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:08.584746 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:08.584806 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:08.587634 (+ 2828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:08.587636 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:08.587636 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:08.587643 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:08.590723 (+ 3080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:08.591288 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:08.591290 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:08.591290 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:08.591354 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:08.591649 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:08.591650 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:08.591650 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:08.591806 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:08.591877 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:08.592198 (+ 321us) server_negotiation.cc:299] Negotiation successful
0928 06:33:08.592283 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:33:08.592971 13418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:08.579400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:36923 (local address 127.0.0.1:56722)
0928 06:33:08.579595 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:08.579652 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:08.580291 (+ 639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:08.580720 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:08.580726 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:08.581149 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:08.581466 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:08.581474 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:08.582593 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:08.582594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:08.584185 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:08.584190 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:08.584917 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:08.584919 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:08.585086 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:08.585726 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:08.585736 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:08.587518 (+ 1782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:08.590832 (+ 3314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:08.590841 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:08.590845 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:08.591186 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:08.591432 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:08.591433 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:08.591433 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:08.591573 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:08.591879 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:08.591887 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:08.592051 (+ 164us) client_negotiation.cc:769] Sending connection context
0928 06:33:08.592176 (+ 125us) client_negotiation.cc:240] Negotiation successful
0928 06:33:08.592338 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":93,"threads_started":1}
I20260928 06:33:08.593886 13416 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.225.189" port: 37297 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:33:08.596622 13420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:08.583701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34880 (local address 127.22.225.188:35927)
0928 06:33:08.583883 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:08.583888 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:08.583910 (+ 22us) server_negotiation.cc:407] Connection header received
0928 06:33:08.583934 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:08.583936 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:08.583959 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:08.584941 (+ 982us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:08.584996 (+ 55us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:08.586129 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:08.587250 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:08.587720 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:08.588052 (+ 332us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:08.591284 (+ 3232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:08.591291 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:08.591292 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:08.591299 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:08.595001 (+ 3702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:08.595394 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:08.595397 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:08.595397 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:08.595453 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:08.595655 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:08.595656 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:08.595656 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:08.595823 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:08.596340 (+ 517us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:08.596386 (+ 46us) server_negotiation.cc:299] Negotiation successful
0928 06:33:08.596461 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":88,"threads_started":1}
Sep 28 06:33:08 dist-test-slave-d59m krb5kdc[13185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577188, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.188@KRBTEST.COM for kudu/127.22.225.189@KRBTEST.COM
Sep 28 06:33:08 dist-test-slave-d59m krb5kdc[13185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:33:08.605674 13346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:08.594389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46212 (local address 127.22.225.189:37297)
0928 06:33:08.594441 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:08.594443 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:08.594453 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:33:08.594490 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:08.594491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:08.594502 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:08.594568 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:08.594936 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:08.595691 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:08.596656 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:08.596933 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:08.596980 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:08.600958 (+ 3978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:08.600960 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:08.600960 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:08.600968 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:08.604319 (+ 3351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:08.604801 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:08.604802 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:08.604802 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:08.604868 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:08.605057 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:08.605058 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:08.605058 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:08.605209 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:08.605275 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:08.605477 (+ 202us) server_negotiation.cc:299] Negotiation successful
0928 06:33:08.605560 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 06:33:08.605988 13418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:08.594256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.189:37297 (local address 127.0.0.1:46212)
0928 06:33:08.594320 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:08.594332 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:08.594381 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:08.594586 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:08.594587 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:08.594779 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:08.594827 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:08.594829 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:08.595779 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:08.595780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:08.596518 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:08.596520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:08.597375 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:08.597376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:08.597443 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:08.599054 (+ 1611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:08.599059 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:08.600846 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:08.604430 (+ 3584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:08.604432 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:08.604432 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:08.604698 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:08.604930 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:08.604931 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:08.604931 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:08.604979 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:08.605291 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:08.605292 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:08.605343 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 06:33:08.605761 (+ 418us) client_negotiation.cc:240] Negotiation successful
0928 06:33:08.605872 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 06:33:08.606518 13416 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.225.188" port: 35927 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:33:08.607501 13269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:08.597547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56732 (local address 127.22.225.190:36923)
0928 06:33:08.597587 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:08.597589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:08.597598 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:33:08.597678 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:08.597679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:08.597689 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:08.597718 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:08.598338 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:08.598895 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:08.600567 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:08.600933 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:08.600972 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:08.603551 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:08.603553 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:08.603553 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:08.603562 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:08.606030 (+ 2468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:08.606377 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:08.606378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:08.606379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:08.606441 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:08.606757 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:08.606758 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:08.606759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:08.606907 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:08.607117 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:08.607308 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 06:33:08.607384 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
Sep 28 06:33:08 dist-test-slave-d59m krb5kdc[13185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577188, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.188@KRBTEST.COM for kudu/127.22.225.188@KRBTEST.COM
Sep 28 06:33:08 dist-test-slave-d59m krb5kdc[13185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.225.189@KRBTEST.COM
I20260928 06:33:08.613548 13418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:08.606799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.188:35927 (local address 127.0.0.1:34894)
0928 06:33:08.606864 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:08.606875 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:08.606912 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:08.607032 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:08.607033 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:08.607197 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:08.607242 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:08.607244 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:08.608128 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:08.608129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:08.608826 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:08.608828 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:08.609230 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:08.609231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:08.609279 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:08.609719 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:08.609724 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:08.610989 (+ 1265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:08.612470 (+ 1481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:08.612471 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:08.612471 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:08.612697 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:08.612897 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:08.612898 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:08.612898 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:08.612977 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:08.613233 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:08.613235 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:08.613285 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 06:33:08.613377 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 06:33:08.613469 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 06:33:08.613680 13420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:08.606885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34894 (local address 127.22.225.188:35927)
0928 06:33:08.606919 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:08.606921 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:08.606931 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:33:08.606953 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:08.606954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:08.606966 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:08.607017 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:08.607324 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:08.608029 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:08.608922 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:08.609143 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:08.609232 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:08.611054 (+ 1822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:08.611055 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:08.611055 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:08.611062 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:08.612386 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:08.612768 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:08.612769 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:08.612769 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:08.612827 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:08.613050 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:08.613051 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:08.613051 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:08.613160 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:08.613209 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:08.613423 (+ 214us) server_negotiation.cc:299] Negotiation successful
0928 06:33:08.613532 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":16}
I20260928 06:33:08.616703 13346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:08.608304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46224 (local address 127.22.225.189:37297)
0928 06:33:08.608362 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:08.608364 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:08.608375 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:33:08.608399 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:08.608400 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:08.608410 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:08.608465 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:08.609058 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:08.609525 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:08.610406 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:08.610703 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:08.610774 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:08.613417 (+ 2643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:08.613418 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:08.613418 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:08.613426 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:08.615361 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:08.615720 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:08.615720 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:08.615721 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:08.615798 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:08.616026 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:08.616027 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:08.616027 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:08.616166 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:08.616353 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:08.616453 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 06:33:08.616527 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 06:33:08.617003 13416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bcb789fca4e421f8da9f41b31334609: Bootstrap starting.
I20260928 06:33:08.618170 13416 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6bcb789fca4e421f8da9f41b31334609: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:08.618394 23430 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:33:08.618525 13416 log.cc:824] T 00000000000000000000000000000000 P 6bcb789fca4e421f8da9f41b31334609: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:08.618868 13341 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.225.188" port: 35927 } attempt: 3
I20260928 06:33:08.619427 13416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bcb789fca4e421f8da9f41b31334609: No bootstrap required, opened a new log
I20260928 06:33:08.619746 13240 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "83509909ca404cc5b5edbd57811dd113"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56732
I20260928 06:33:08.621590 13416 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6bcb789fca4e421f8da9f41b31334609 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83509909ca404cc5b5edbd57811dd113" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36923 } } peers { permanent_uuid: "d5f2a3245b594c5ca7b7fc017ce03df4" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 37297 } } peers { permanent_uuid: "6bcb789fca4e421f8da9f41b31334609" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 35927 } }
I20260928 06:33:08.621755 13416 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6bcb789fca4e421f8da9f41b31334609 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bcb789fca4e421f8da9f41b31334609, State: Initialized, Role: FOLLOWER
I20260928 06:33:08.621848 13416 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6bcb789fca4e421f8da9f41b31334609 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83509909ca404cc5b5edbd57811dd113" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36923 } } peers { permanent_uuid: "d5f2a3245b594c5ca7b7fc017ce03df4" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 37297 } } peers { permanent_uuid: "6bcb789fca4e421f8da9f41b31334609" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 35927 } }
I20260928 06:33:08.622182 13416 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6bcb789fca4e421f8da9f41b31334609 [sys.catalog]: configured and running, proceeding with master startup.
Sep 28 06:33:08 dist-test-slave-d59m krb5kdc[13185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577188, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.189@KRBTEST.COM for kudu/127.22.225.188@KRBTEST.COM
W20260928 06:33:08.624039 13433 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6bcb789fca4e421f8da9f41b31334609: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:33:08.624200 13433 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:33:08.623603 13422 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bcb789fca4e421f8da9f41b31334609 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83509909ca404cc5b5edbd57811dd113" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36923 } } peers { permanent_uuid: "d5f2a3245b594c5ca7b7fc017ce03df4" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 37297 } } peers { permanent_uuid: "6bcb789fca4e421f8da9f41b31334609" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 35927 } } }
I20260928 06:33:08.626974 13343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:08.619122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.188:35927 (local address 127.0.0.1:34910)
0928 06:33:08.619169 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:08.619180 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:08.619217 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:08.619356 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:08.619357 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:08.619529 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:08.619583 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:08.619585 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:08.620471 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:08.620472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:08.621216 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:08.621219 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:08.621688 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:08.621689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:08.621741 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:08.622190 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:08.622196 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:08.623639 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:08.625614 (+ 1975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:08.625615 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:08.625616 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:08.625832 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:08.626119 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:08.626120 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:08.626120 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:08.626212 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:08.626641 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:08.626642 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:08.626706 (+ 64us) client_negotiation.cc:769] Sending connection context
0928 06:33:08.626773 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 06:33:08.626877 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 06:33:08.627285 13420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:08.619206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34910 (local address 127.22.225.188:35927)
0928 06:33:08.619247 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:08.619249 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:08.619258 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:33:08.619278 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:08.619279 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:08.619287 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:08.619352 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:08.619661 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:08.620367 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:08.621305 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:08.621602 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:08.621671 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:08.623713 (+ 2042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:08.623714 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:08.623715 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:08.623721 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:08.625473 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:08.625943 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:08.625945 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:08.625945 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:08.626019 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:08.626316 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:08.626317 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:08.626318 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:08.626523 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:08.626598 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:08.627127 (+ 529us) server_negotiation.cc:299] Negotiation successful
0928 06:33:08.627200 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:33:08.627544 13422 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bcb789fca4e421f8da9f41b31334609 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:33:08.624940 13348 master_runner.cc:186] Error getting master registration for 127.22.225.190:36923: OK, instance_id { permanent_uuid: "83509909ca404cc5b5edbd57811dd113" instance_seqno: 1790577188202259 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 06:33:08.628217 13348 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 06:33:08.629336 13341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5f2a3245b594c5ca7b7fc017ce03df4: Bootstrap starting.
I20260928 06:33:08.630777 13341 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5f2a3245b594c5ca7b7fc017ce03df4: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:08.631120 13341 log.cc:824] T 00000000000000000000000000000000 P d5f2a3245b594c5ca7b7fc017ce03df4: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:08.632500 13341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5f2a3245b594c5ca7b7fc017ce03df4: No bootstrap required, opened a new log
I20260928 06:33:08.634677 13341 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5f2a3245b594c5ca7b7fc017ce03df4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83509909ca404cc5b5edbd57811dd113" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36923 } } peers { permanent_uuid: "d5f2a3245b594c5ca7b7fc017ce03df4" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 37297 } } peers { permanent_uuid: "6bcb789fca4e421f8da9f41b31334609" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 35927 } }
I20260928 06:33:08.634830 13341 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5f2a3245b594c5ca7b7fc017ce03df4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5f2a3245b594c5ca7b7fc017ce03df4, State: Initialized, Role: FOLLOWER
I20260928 06:33:08.634918 13341 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5f2a3245b594c5ca7b7fc017ce03df4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83509909ca404cc5b5edbd57811dd113" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36923 } } peers { permanent_uuid: "d5f2a3245b594c5ca7b7fc017ce03df4" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 37297 } } peers { permanent_uuid: "6bcb789fca4e421f8da9f41b31334609" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 35927 } }
I20260928 06:33:08.635197 13436 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5f2a3245b594c5ca7b7fc017ce03df4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83509909ca404cc5b5edbd57811dd113" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36923 } } peers { permanent_uuid: "d5f2a3245b594c5ca7b7fc017ce03df4" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 37297 } } peers { permanent_uuid: "6bcb789fca4e421f8da9f41b31334609" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 35927 } } }
I20260928 06:33:08.635277 13341 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5f2a3245b594c5ca7b7fc017ce03df4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:33:08.635299 13436 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5f2a3245b594c5ca7b7fc017ce03df4 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 06:33:08.637432 13447 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d5f2a3245b594c5ca7b7fc017ce03df4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:33:08.637544 13447 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:33:08.637549 13273 master_runner.cc:186] Error getting master registration for 127.22.225.190:36923: OK, instance_id { permanent_uuid: "83509909ca404cc5b5edbd57811dd113" instance_seqno: 1790577188202259 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 06:33:08.640544 13273 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 06:33:08.654331 13266 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.225.188" port: 35927 } attempt: 3
Sep 28 06:33:08 dist-test-slave-d59m krb5kdc[13185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577188, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.190@KRBTEST.COM for kudu/127.22.225.188@KRBTEST.COM
I20260928 06:33:08.663930 13270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:08.654759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.188:35927 (local address 127.0.0.1:34924)
0928 06:33:08.654829 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:08.654842 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:08.654894 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:08.655258 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:08.655260 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:08.655463 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:08.655522 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:08.655524 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:08.656503 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:08.656504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:08.657354 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:08.657357 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:08.657865 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:08.657866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:08.657937 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:08.658725 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:08.658731 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:08.660280 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:08.662720 (+ 2440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:08.662722 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:08.662722 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:08.662949 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:08.663199 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:08.663200 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:08.663200 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:08.663271 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:08.663588 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:08.663590 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:08.663656 (+ 66us) client_negotiation.cc:769] Sending connection context
0928 06:33:08.663718 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 06:33:08.663819 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 06:33:08.664422 13420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:08.654988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34924 (local address 127.22.225.188:35927)
0928 06:33:08.655049 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:08.655052 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:08.655066 (+ 14us) server_negotiation.cc:407] Connection header received
0928 06:33:08.655098 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:08.655099 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:08.655111 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:08.655193 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:08.655608 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:08.656418 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:08.657464 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:08.657785 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:08.657854 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:08.660372 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:08.660373 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:08.660374 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:08.660382 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:08.662601 (+ 2219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:08.663046 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:08.663047 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:08.663048 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:08.663118 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:08.663344 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:08.663345 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:08.663345 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:08.663490 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:08.663549 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:08.664251 (+ 702us) server_negotiation.cc:299] Negotiation successful
0928 06:33:08.664333 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 06:33:08.666249 13266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83509909ca404cc5b5edbd57811dd113: Bootstrap starting.
I20260928 06:33:08.666991 13266 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83509909ca404cc5b5edbd57811dd113: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:08.667339 13266 log.cc:824] T 00000000000000000000000000000000 P 83509909ca404cc5b5edbd57811dd113: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:08.668283 13266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83509909ca404cc5b5edbd57811dd113: No bootstrap required, opened a new log
I20260928 06:33:08.672043 13266 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83509909ca404cc5b5edbd57811dd113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83509909ca404cc5b5edbd57811dd113" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36923 } } peers { permanent_uuid: "d5f2a3245b594c5ca7b7fc017ce03df4" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 37297 } } peers { permanent_uuid: "6bcb789fca4e421f8da9f41b31334609" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 35927 } }
I20260928 06:33:08.672199 13266 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83509909ca404cc5b5edbd57811dd113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83509909ca404cc5b5edbd57811dd113, State: Initialized, Role: FOLLOWER
I20260928 06:33:08.672335 13266 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83509909ca404cc5b5edbd57811dd113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83509909ca404cc5b5edbd57811dd113" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36923 } } peers { permanent_uuid: "d5f2a3245b594c5ca7b7fc017ce03df4" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 37297 } } peers { permanent_uuid: "6bcb789fca4e421f8da9f41b31334609" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 35927 } }
I20260928 06:33:08.672775 13266 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83509909ca404cc5b5edbd57811dd113 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:33:08.674340 13240 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83509909ca404cc5b5edbd57811dd113 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 06:33:08.674736 13240 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83509909ca404cc5b5edbd57811dd113 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:08.675494 13240 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83509909ca404cc5b5edbd57811dd113 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83509909ca404cc5b5edbd57811dd113" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36923 } } peers { permanent_uuid: "d5f2a3245b594c5ca7b7fc017ce03df4" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 37297 } } peers { permanent_uuid: "6bcb789fca4e421f8da9f41b31334609" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 35927 } }
I20260928 06:33:08.675803 13240 leader_election.cc:288] T 00000000000000000000000000000000 P 83509909ca404cc5b5edbd57811dd113 [CANDIDATE]: Term 1 election: Requested vote from peers d5f2a3245b594c5ca7b7fc017ce03df4 (127.22.225.189:37297), 6bcb789fca4e421f8da9f41b31334609 (127.22.225.188:35927)
I20260928 06:33:08.676153 13315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "83509909ca404cc5b5edbd57811dd113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d5f2a3245b594c5ca7b7fc017ce03df4"
I20260928 06:33:08.676297 13315 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5f2a3245b594c5ca7b7fc017ce03df4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:08.676326 13390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "83509909ca404cc5b5edbd57811dd113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6bcb789fca4e421f8da9f41b31334609"
I20260928 06:33:08.676462 13390 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6bcb789fca4e421f8da9f41b31334609 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:08.677213 13315 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d5f2a3245b594c5ca7b7fc017ce03df4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83509909ca404cc5b5edbd57811dd113 in term 1.
I20260928 06:33:08.677453 13216 leader_election.cc:302] T 00000000000000000000000000000000 P 83509909ca404cc5b5edbd57811dd113 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83509909ca404cc5b5edbd57811dd113, d5f2a3245b594c5ca7b7fc017ce03df4; no voters:
I20260928 06:33:08.677945 13390 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6bcb789fca4e421f8da9f41b31334609 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83509909ca404cc5b5edbd57811dd113 in term 1.
W20260928 06:33:08.679252 13461 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 83509909ca404cc5b5edbd57811dd113: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260928 06:33:08.679688 13198 master_runner.cc:263] No leader master found from master 83509909ca404cc5b5edbd57811dd113
I20260928 06:33:08.679811 13198 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 06:33:08.679896 13461 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:33:08.681216 13467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83509909ca404cc5b5edbd57811dd113 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83509909ca404cc5b5edbd57811dd113" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36923 } } peers { permanent_uuid: "d5f2a3245b594c5ca7b7fc017ce03df4" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 37297 } } peers { permanent_uuid: "6bcb789fca4e421f8da9f41b31334609" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 35927 } } }
I20260928 06:33:08.681358 13467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83509909ca404cc5b5edbd57811dd113 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:33:08.681450 13467 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83509909ca404cc5b5edbd57811dd113 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:33:08.681537 13467 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83509909ca404cc5b5edbd57811dd113 [term 1 LEADER]: Becoming Leader. State: Replica: 83509909ca404cc5b5edbd57811dd113, State: Running, Role: LEADER
I20260928 06:33:08.681669 13467 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83509909ca404cc5b5edbd57811dd113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83509909ca404cc5b5edbd57811dd113" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36923 } } peers { permanent_uuid: "d5f2a3245b594c5ca7b7fc017ce03df4" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 37297 } } peers { permanent_uuid: "6bcb789fca4e421f8da9f41b31334609" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 35927 } }
I20260928 06:33:08.682168 13467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83509909ca404cc5b5edbd57811dd113 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83509909ca404cc5b5edbd57811dd113. Latest consensus state: current_term: 1 leader_uuid: "83509909ca404cc5b5edbd57811dd113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83509909ca404cc5b5edbd57811dd113" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36923 } } peers { permanent_uuid: "d5f2a3245b594c5ca7b7fc017ce03df4" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 37297 } } peers { permanent_uuid: "6bcb789fca4e421f8da9f41b31334609" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 35927 } } }
I20260928 06:33:08.682281 13467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83509909ca404cc5b5edbd57811dd113 [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:08.690040 13346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:08.680467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46240 (local address 127.22.225.189:37297)
0928 06:33:08.680519 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:08.680522 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:08.681037 (+ 515us) server_negotiation.cc:407] Connection header received
0928 06:33:08.681153 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:08.681154 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:08.681165 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:08.681194 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:08.682182 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:08.682974 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:08.684149 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:08.684451 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:08.684491 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:08.687010 (+ 2519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:08.687011 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:08.687011 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:08.687019 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:08.688794 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:08.689181 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:08.689182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:08.689183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:08.689239 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:08.689432 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:08.689432 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:08.689433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:08.689569 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:08.689651 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:08.689861 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 06:33:08.689939 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 06:33:08.697332 13472 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:33:08.697552 13472 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:33:08.699010 13420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:08.680216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34932 (local address 127.22.225.188:35927)
0928 06:33:08.680264 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:08.680266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:08.684932 (+ 4666us) server_negotiation.cc:407] Connection header received
0928 06:33:08.685002 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:08.685003 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:08.685016 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:08.685078 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:08.685724 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:08.686521 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:08.690292 (+ 3771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:08.690698 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:08.690747 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:08.693660 (+ 2913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:08.693662 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:08.693662 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:08.693672 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:08.696806 (+ 3134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:08.697444 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:08.697445 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:08.697445 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:08.697500 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:08.698001 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:08.698002 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:08.698002 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:08.698134 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:08.698217 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:08.698820 (+ 603us) server_negotiation.cc:299] Negotiation successful
0928 06:33:08.698905 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 06:33:08.699661 13219 tablet.cc:2369] T 00000000000000000000000000000000 P 83509909ca404cc5b5edbd57811dd113: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:33:08.700143 13269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:08.680690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56736 (local address 127.22.225.190:36923)
0928 06:33:08.680738 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:08.680740 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:08.682535 (+ 1795us) server_negotiation.cc:407] Connection header received
0928 06:33:08.682649 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:08.682650 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:08.682662 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:08.682695 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:08.683531 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:08.684387 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:08.687816 (+ 3429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:08.688176 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:08.688224 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:08.693966 (+ 5742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:08.693967 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:08.693968 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:08.693977 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:08.696098 (+ 2121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:08.697681 (+ 1583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:08.697682 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:08.697683 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:08.697739 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:08.698402 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:08.698403 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:08.698403 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:08.698538 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:08.698624 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:08.699967 (+ 1343us) server_negotiation.cc:299] Negotiation successful
0928 06:33:08.700051 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:33:08.712891 13390 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6bcb789fca4e421f8da9f41b31334609 [term 1 FOLLOWER]: Refusing update from remote peer 83509909ca404cc5b5edbd57811dd113: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:33:08.712963 13315 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d5f2a3245b594c5ca7b7fc017ce03df4 [term 1 FOLLOWER]: Refusing update from remote peer 83509909ca404cc5b5edbd57811dd113: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:33:08.713263 13450 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 83509909ca404cc5b5edbd57811dd113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bcb789fca4e421f8da9f41b31334609" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 35927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:33:08.713433 13467 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 83509909ca404cc5b5edbd57811dd113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5f2a3245b594c5ca7b7fc017ce03df4" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 37297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:33:08.714983 13436 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5f2a3245b594c5ca7b7fc017ce03df4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83509909ca404cc5b5edbd57811dd113. Latest consensus state: current_term: 1 leader_uuid: "83509909ca404cc5b5edbd57811dd113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83509909ca404cc5b5edbd57811dd113" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36923 } } peers { permanent_uuid: "d5f2a3245b594c5ca7b7fc017ce03df4" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 37297 } } peers { permanent_uuid: "6bcb789fca4e421f8da9f41b31334609" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 35927 } } }
I20260928 06:33:08.715066 13436 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5f2a3245b594c5ca7b7fc017ce03df4 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:33:08.716097 13422 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bcb789fca4e421f8da9f41b31334609 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83509909ca404cc5b5edbd57811dd113. Latest consensus state: current_term: 1 leader_uuid: "83509909ca404cc5b5edbd57811dd113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83509909ca404cc5b5edbd57811dd113" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36923 } } peers { permanent_uuid: "d5f2a3245b594c5ca7b7fc017ce03df4" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 37297 } } peers { permanent_uuid: "6bcb789fca4e421f8da9f41b31334609" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 35927 } } }
I20260928 06:33:08.716180 13422 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bcb789fca4e421f8da9f41b31334609 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:33:08.722482 13450 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83509909ca404cc5b5edbd57811dd113 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "83509909ca404cc5b5edbd57811dd113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83509909ca404cc5b5edbd57811dd113" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36923 } } peers { permanent_uuid: "d5f2a3245b594c5ca7b7fc017ce03df4" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 37297 } } peers { permanent_uuid: "6bcb789fca4e421f8da9f41b31334609" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 35927 } } }
I20260928 06:33:08.722607 13450 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83509909ca404cc5b5edbd57811dd113 [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:08.722692 13473 mvcc.cc:205] Tried to move back new op lower bound from 7334204164964986880 to 7334204164841345024. Current Snapshot: MvccSnapshot[applied={T|T < 7334204164964986880}]
I20260928 06:33:08.723279 13422 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bcb789fca4e421f8da9f41b31334609 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "83509909ca404cc5b5edbd57811dd113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83509909ca404cc5b5edbd57811dd113" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36923 } } peers { permanent_uuid: "d5f2a3245b594c5ca7b7fc017ce03df4" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 37297 } } peers { permanent_uuid: "6bcb789fca4e421f8da9f41b31334609" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 35927 } } }
I20260928 06:33:08.723361 13422 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bcb789fca4e421f8da9f41b31334609 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:33:08.723659 13476 mvcc.cc:205] Tried to move back new op lower bound from 7334204164964986880 to 7334204164841345024. Current Snapshot: MvccSnapshot[applied={T|T < 7334204164964986880}]
I20260928 06:33:08.724014 13467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83509909ca404cc5b5edbd57811dd113 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "83509909ca404cc5b5edbd57811dd113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83509909ca404cc5b5edbd57811dd113" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36923 } } peers { permanent_uuid: "d5f2a3245b594c5ca7b7fc017ce03df4" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 37297 } } peers { permanent_uuid: "6bcb789fca4e421f8da9f41b31334609" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 35927 } } }
I20260928 06:33:08.724093 13467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83509909ca404cc5b5edbd57811dd113 [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:08.724437 13436 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5f2a3245b594c5ca7b7fc017ce03df4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "83509909ca404cc5b5edbd57811dd113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83509909ca404cc5b5edbd57811dd113" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 36923 } } peers { permanent_uuid: "d5f2a3245b594c5ca7b7fc017ce03df4" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 37297 } } peers { permanent_uuid: "6bcb789fca4e421f8da9f41b31334609" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 35927 } } }
I20260928 06:33:08.724520 13436 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5f2a3245b594c5ca7b7fc017ce03df4 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:33:08.725247 13474 mvcc.cc:205] Tried to move back new op lower bound from 7334204164964986880 to 7334204164841345024. Current Snapshot: MvccSnapshot[applied={T|T < 7334204164964986880}]
I20260928 06:33:08.733283 13472 catalog_manager.cc:1347] Generated new cluster ID: 8f8afcca06914942bfc37e2d0c21908b
I20260928 06:33:08.733358 13472 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:33:08.743486 13472 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:33:08.744283 13472 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:33:08.757017 13472 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83509909ca404cc5b5edbd57811dd113: Generated new TSK 0
I20260928 06:33:08.757249 13472 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 06:33:08.794234 13420 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:33:08.787250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34940 (local address 127.22.225.188:35927)
0928 06:33:08.787303 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:08.787305 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:08.793546 (+ 6241us) server_negotiation.cc:407] Connection header received
0928 06:33:08.793635 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:08.793637 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:08.793649 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:08.793709 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:08.794142 (+ 433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34940: BlockingRecv error: recv got EOF from 127.0.0.1:34940 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260928 06:33:08.794749 13269 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:33:08.793139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56752 (local address 127.22.225.190:36923)
0928 06:33:08.793202 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:08.793204 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:08.794186 (+ 982us) server_negotiation.cc:407] Connection header received
0928 06:33:08.794256 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:08.794257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:08.794270 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:08.794364 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:08.794679 (+ 315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56752: BlockingRecv error: recv got EOF from 127.0.0.1:56752 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260928 06:33:08.795096 13346 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:33:08.792917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46256 (local address 127.22.225.189:37297)
0928 06:33:08.792967 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:08.792969 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:08.794638 (+ 1669us) server_negotiation.cc:407] Connection header received
0928 06:33:08.794707 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:08.794708 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:08.794719 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:08.794777 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:08.795036 (+ 259us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46256: BlockingRecv error: recv got EOF from 127.0.0.1:46256 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260928 06:33:08.808801 13346 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:33:08.807796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46268 (local address 127.22.225.189:37297)
0928 06:33:08.807847 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:08.807849 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:08.808123 (+ 274us) server_negotiation.cc:407] Connection header received
0928 06:33:08.808243 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:08.808244 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:08.808266 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:08.808297 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:08.808738 (+ 441us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46268: BlockingRecv error: recv got EOF from 127.0.0.1:46268 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 06:33:08.837371 13346 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:33:08.817015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46278 (local address 127.22.225.189:37297)
0928 06:33:08.817058 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:08.817060 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:08.835830 (+ 18770us) server_negotiation.cc:407] Connection header received
0928 06:33:08.835915 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:08.835917 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:08.835933 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:08.836014 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:08.837307 (+ 1293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46278: BlockingRecv error: recv got EOF from 127.0.0.1:46278 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 06:33:08.852836 13346 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:33:08.843828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46282 (local address 127.22.225.189:37297)
0928 06:33:08.843882 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:08.843885 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:08.848814 (+ 4929us) server_negotiation.cc:407] Connection header received
0928 06:33:08.852213 (+ 3399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:08.852215 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:08.852233 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:08.852326 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:08.852756 (+ 430us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46282: BlockingRecv error: recv got EOF from 127.0.0.1:46282 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260928 06:33:08.870899 13346 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:33:08.857900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46290 (local address 127.22.225.189:37297)
0928 06:33:08.857959 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:08.857961 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:08.870211 (+ 12250us) server_negotiation.cc:407] Connection header received
0928 06:33:08.870256 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:08.870257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:08.870274 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:08.870342 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:08.870833 (+ 491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46290: BlockingRecv error: recv got EOF from 127.0.0.1:46290 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260928 06:33:08.897469 13346 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:33:08.884984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46300 (local address 127.22.225.189:37297)
0928 06:33:08.885038 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:08.885041 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:08.896832 (+ 11791us) server_negotiation.cc:407] Connection header received
0928 06:33:08.896952 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:08.896954 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:08.896968 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:08.896999 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:08.897402 (+ 403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46300: BlockingRecv error: recv got EOF from 127.0.0.1:46300 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260928 06:33:08.925522 13346 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:33:08.917981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46314 (local address 127.22.225.189:37297)
0928 06:33:08.918035 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:08.918037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:08.924820 (+ 6783us) server_negotiation.cc:407] Connection header received
0928 06:33:08.924942 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:08.924944 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:08.924957 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:08.924989 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:08.925437 (+ 448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46314: BlockingRecv error: recv got EOF from 127.0.0.1:46314 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260928 06:33:08.959986 13346 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:33:08.959118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46316 (local address 127.22.225.189:37297)
0928 06:33:08.959179 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:08.959181 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:08.959316 (+ 135us) server_negotiation.cc:407] Connection header received
0928 06:33:08.959362 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:08.959363 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:08.959376 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:08.959441 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:08.959907 (+ 466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46316: BlockingRecv error: recv got EOF from 127.0.0.1:46316 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260928 06:33:09.026057 13346 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:33:09.025197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46320 (local address 127.22.225.189:37297)
0928 06:33:09.025252 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:09.025254 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:09.025337 (+ 83us) server_negotiation.cc:407] Connection header received
0928 06:33:09.025432 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:09.025433 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:09.025447 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:09.025503 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:09.025979 (+ 476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46320: BlockingRecv error: recv got EOF from 127.0.0.1:46320 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260928 06:33:09.156631 13346 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:33:09.155774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46330 (local address 127.22.225.189:37297)
0928 06:33:09.155832 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:09.155834 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:09.155971 (+ 137us) server_negotiation.cc:407] Connection header received
0928 06:33:09.156036 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:09.156037 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:09.156051 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:09.156132 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:09.156528 (+ 396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46330: BlockingRecv error: recv got EOF from 127.0.0.1:46330 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260928 06:33:09.414788 13346 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:33:09.413977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46342 (local address 127.22.225.189:37297)
0928 06:33:09.414031 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:09.414034 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:09.414088 (+ 54us) server_negotiation.cc:407] Connection header received
0928 06:33:09.414197 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:09.414198 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:09.414212 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:09.414311 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:09.414697 (+ 386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46342: BlockingRecv error: recv got EOF from 127.0.0.1:46342 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 06:33:09.625208 13433 catalog_manager.cc:1259] Loaded cluster ID: 8f8afcca06914942bfc37e2d0c21908b
I20260928 06:33:09.625296 13433 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6bcb789fca4e421f8da9f41b31334609: loading cluster ID for follower catalog manager: success
I20260928 06:33:09.626163 13433 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6bcb789fca4e421f8da9f41b31334609: acquiring CA information for follower catalog manager: success
I20260928 06:33:09.626497 13433 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6bcb789fca4e421f8da9f41b31334609: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 06:33:09.638160 13447 catalog_manager.cc:1259] Loaded cluster ID: 8f8afcca06914942bfc37e2d0c21908b
I20260928 06:33:09.638264 13447 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d5f2a3245b594c5ca7b7fc017ce03df4: loading cluster ID for follower catalog manager: success
I20260928 06:33:09.639118 13447 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d5f2a3245b594c5ca7b7fc017ce03df4: acquiring CA information for follower catalog manager: success
I20260928 06:33:09.639446 13447 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d5f2a3245b594c5ca7b7fc017ce03df4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 06:33:09.931113 13545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:09.927914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46348 (local address 127.22.225.189:37297)
0928 06:33:09.928053 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:09.928055 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:09.928171 (+ 116us) server_negotiation.cc:407] Connection header received
0928 06:33:09.928272 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:09.928273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:09.928340 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:09.928422 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:33:09.928679 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:09.929198 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:09.930064 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:09.930369 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:09.930521 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:09.930720 (+ 199us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:33:09.930841 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:33:09.930932 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:09.930974 (+ 42us) server_negotiation.cc:299] Negotiation successful
0928 06:33:09.930998 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":74,"threads_started":1}
I20260928 06:33:09.936230 13551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:09.933187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34944 (local address 127.22.225.188:35927)
0928 06:33:09.933290 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:09.933292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:09.933302 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:33:09.933350 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:09.933351 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:09.933399 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:09.933488 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:33:09.933579 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:09.934326 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:09.935123 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:09.935450 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:09.935524 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:09.935749 (+ 225us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:33:09.935889 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:33:09.935978 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:09.935998 (+ 20us) server_negotiation.cc:299] Negotiation successful
0928 06:33:09.936022 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":59,"threads_started":1}
I20260928 06:33:09.941792 13551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:09.938261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34960 (local address 127.22.225.188:35927)
0928 06:33:09.938323 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:09.938325 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:09.938462 (+ 137us) server_negotiation.cc:407] Connection header received
0928 06:33:09.938490 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:09.938491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:09.938502 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:09.938556 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:33:09.938899 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:09.939653 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:09.940892 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:09.941166 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:09.941251 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:09.941410 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:33:09.941444 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:33:09.941492 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:09.941714 (+ 222us) server_negotiation.cc:299] Negotiation successful
0928 06:33:09.941734 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 06:33:09.941779 13545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:09.938078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46354 (local address 127.22.225.189:37297)
0928 06:33:09.938119 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:09.938121 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:09.938573 (+ 452us) server_negotiation.cc:407] Connection header received
0928 06:33:09.938648 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:09.938650 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:09.938664 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:09.938726 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:33:09.938884 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:09.939632 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:09.940957 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:09.941243 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:09.941314 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:09.941489 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:33:09.941526 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:33:09.941582 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:09.941678 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 06:33:09.941705 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 06:33:09.942878 13558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:09.938407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56762 (local address 127.22.225.190:36923)
0928 06:33:09.938528 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:09.938530 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:09.938772 (+ 242us) server_negotiation.cc:407] Connection header received
0928 06:33:09.938825 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:09.938826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:09.938883 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:09.938958 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:33:09.939141 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:09.939859 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:09.941545 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:09.942330 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:09.942405 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:09.942554 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:33:09.942668 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:33:09.942728 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:09.942773 (+ 45us) server_negotiation.cc:299] Negotiation successful
0928 06:33:09.942795 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":62,"threads_started":1}
I20260928 06:33:09.944679 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 13198
I20260928 06:33:09.951069 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 13273
I20260928 06:33:09.956638 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 13348
2026-09-28T06:33:09Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3194 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:33:10 dist-test-slave-d59m krb5kdc[13562](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:33:10 dist-test-slave-d59m krb5kdc[13562](info): set up 2 sockets
Sep 28 06:33:10 dist-test-slave-d59m krb5kdc[13562](info): commencing operation
krb5kdc: starting...
W20260928 06:33:11.001487 23430 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/krb5kdc/test-user.keytab.
Sep 28 06:33:11 dist-test-slave-d59m krb5kdc[13562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:33:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:33:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab.
Entry for principal kudu/127.22.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab.
I20260928 06:33:11.128535 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:46789
--webserver_interface=127.22.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab
--principal=kudu/127.22.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:39585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.225.190:46789,127.22.225.189:37899,127.22.225.188:34885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:11.212407 13575 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 06:33:11.212733 13575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:11.212778 13575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:11.212823 13575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:11.214053 13575 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:33:11.214125 13575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:11.214150 13575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:11.214171 13575 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:33:11.214200 13575 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:33:11.215842 13575 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:39585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.225.190:46789,127.22.225.189:37899,127.22.225.188:34885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.190.keytab
--principal=kudu/127.22.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.190:46789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13575
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:11.216140 13575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:11.216377 13575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:11.222239 13581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:11.222267 13580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:11.222563 13583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:11.222589 13575 server_base.cc:1034] running on GCE node
I20260928 06:33:11.222882 13575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:11.223201 13575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:11.224381 13575 hybrid_clock.cc:648] HybridClock initialized: now 1790577191224350 us; error 47 us; skew 500 ppm
Sep 28 06:33:11 dist-test-slave-d59m krb5kdc[13562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577191, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:11.227161 13575 init.cc:377] Logged in from keytab as kudu/127.22.225.190@KRBTEST.COM (short username kudu)
I20260928 06:33:11.227669 13575 webserver.cc:460] Webserver started at http://127.22.225.190:35215/ using document root <none> and password file <none>
I20260928 06:33:11.227898 13575 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:11.227967 13575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:11.228123 13575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:11.229319 13575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/minicluster-data/master-0/data/instance:
uuid: "d367f41741664aefbf4679dec7413564"
format_stamp: "Formatted at 2026-09-28 06:33:11 on dist-test-slave-d59m"
server_key: "5a1913448c2a100f3bb281c0959c52d2"
server_key_iv: "b7d09422ba424c599b7b7b241409e167"
server_key_version: "encryptionkey@0"
I20260928 06:33:11.229692 13575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/minicluster-data/master-0/wal/instance:
uuid: "d367f41741664aefbf4679dec7413564"
format_stamp: "Formatted at 2026-09-28 06:33:11 on dist-test-slave-d59m"
server_key: "5a1913448c2a100f3bb281c0959c52d2"
server_key_iv: "b7d09422ba424c599b7b7b241409e167"
server_key_version: "encryptionkey@0"
I20260928 06:33:11.231139 13575 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:33:11.232036 13590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:11.232249 13575 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:11.232345 13575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/minicluster-data/master-0/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/minicluster-data/master-0/wal
uuid: "d367f41741664aefbf4679dec7413564"
format_stamp: "Formatted at 2026-09-28 06:33:11 on dist-test-slave-d59m"
server_key: "5a1913448c2a100f3bb281c0959c52d2"
server_key_iv: "b7d09422ba424c599b7b7b241409e167"
server_key_version: "encryptionkey@0"
I20260928 06:33:11.232415 13575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:11.248072 13575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:11.250685 13575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:11.250833 13575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:11.255546 13575 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.190:46789
I20260928 06:33:11.255620 13642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.190:46789 every 8 connection(s)
I20260928 06:33:11.255971 13575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/minicluster-data/master-0/data/info.pb
I20260928 06:33:11.256963 13643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:11.258555 13643 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.225.190" port: 46789 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:33:11.264248 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 13575
I20260928 06:33:11.264369 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/minicluster-data/master-0/wal/instance
I20260928 06:33:11.264554 23430 external_mini_cluster.cc:1428] Setting key 7033396ea6003a251198abeabfb678f8
Sep 28 06:33:11 dist-test-slave-d59m krb5kdc[13562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577191, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.190@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:33:11.269482 13646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:11.259294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54916 (local address 127.22.225.190:46789)
0928 06:33:11.259518 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:11.259522 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:11.259591 (+ 69us) server_negotiation.cc:407] Connection header received
0928 06:33:11.260145 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:11.260148 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:11.260167 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:11.260276 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:11.260897 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:11.261715 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:11.262604 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:11.262927 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:11.263066 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:11.265589 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:11.265594 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:11.265595 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:11.265601 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:11.267538 (+ 1937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:11.268034 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:11.268037 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:11.268037 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:11.268149 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:11.268435 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:11.268436 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:11.268437 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:11.268745 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:11.268827 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:11.269076 (+ 249us) server_negotiation.cc:299] Negotiation successful
0928 06:33:11.269228 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":201,"thread_start_us":97,"threads_started":1}
I20260928 06:33:11.269526 13645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:11.259272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:46789 (local address 127.0.0.1:54916)
0928 06:33:11.259447 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:11.259492 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:11.259986 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:11.260280 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:11.260283 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:11.260628 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:11.260788 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:11.260793 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:11.261829 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:11.261830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:11.262500 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:11.262501 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:11.263032 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:11.263034 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:11.263101 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:11.263663 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:11.263671 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:11.265494 (+ 1823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:11.267639 (+ 2145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:11.267646 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:11.267648 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:11.267940 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:11.268232 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:11.268234 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:11.268234 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:11.268357 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:11.268825 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:11.268833 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:11.268917 (+ 84us) client_negotiation.cc:769] Sending connection context
0928 06:33:11.269034 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 06:33:11.269324 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":119,"thread_start_us":98,"threads_started":1}
I20260928 06:33:11.270704 13643 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.225.189" port: 37899 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:33:11.271064 13645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:11.270941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.189:37899 (local address 127.0.0.1:48710)
0928 06:33:11.270994 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:11.271042 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.189:37899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 06:33:11.271235 13594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.225.189:37899: connect: Connection refused (error 111)
I20260928 06:33:11.271684 13645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:11.271583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.189:37899 (local address 127.0.0.1:48720)
0928 06:33:11.271629 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:11.271663 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.189:37899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 06:33:11.271811 13643 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.225.189:37899: Network error: Client connection negotiation failed: client connection to 127.22.225.189:37899: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.22.225.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.189.keytab.
Entry for principal kudu/127.22.225.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.189.keytab.
I20260928 06:33:11.292388 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.189:37899
--webserver_interface=127.22.225.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.189.keytab
--principal=kudu/127.22.225.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:39585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.225.190:46789,127.22.225.189:37899,127.22.225.188:34885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:33:11.321062 13643 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.225.189" port: 37899 } attempt: 1
I20260928 06:33:11.321601 13645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:11.321426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.189:37899 (local address 127.0.0.1:48730)
0928 06:33:11.321504 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:11.321554 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.189:37899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 06:33:11.322110 13645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:11.321983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.189:37899 (local address 127.0.0.1:48742)
0928 06:33:11.322044 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:11.322080 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.189:37899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 06:33:11.322207 13643 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.225.189:37899: Network error: Client connection negotiation failed: client connection to 127.22.225.189:37899: connect: Connection refused (error 111)
W20260928 06:33:11.390174 13650 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
I20260928 06:33:11.390355 13643 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.225.189" port: 37899 } attempt: 2
W20260928 06:33:11.390458 13650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:11.390493 13650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:11.390548 13650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 06:33:11.390844 13645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:11.390699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.189:37899 (local address 127.0.0.1:48748)
0928 06:33:11.390764 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:11.390808 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.189:37899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 06:33:11.391227 13645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:11.391130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.189:37899 (local address 127.0.0.1:48752)
0928 06:33:11.391176 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:11.391205 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.189:37899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 06:33:11.391356 13643 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.225.189:37899: Network error: Client connection negotiation failed: client connection to 127.22.225.189:37899: connect: Connection refused (error 111)
W20260928 06:33:11.392066 13650 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:33:11.392113 13650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:11.392128 13650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:11.392160 13650 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:33:11.392189 13650 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:33:11.393837 13650 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:39585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.22.225.190:46789,127.22.225.189:37899,127.22.225.188:34885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.189.keytab
--principal=kudu/127.22.225.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.189:37899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.22.225.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:11.394119 13650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:11.394349 13650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:11.399224 13656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:11.399221 13658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:11.399222 13655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:11.399469 13650 server_base.cc:1034] running on GCE node
I20260928 06:33:11.399703 13650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:11.399938 13650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:11.401085 13650 hybrid_clock.cc:648] HybridClock initialized: now 1790577191401073 us; error 28 us; skew 500 ppm
Sep 28 06:33:11 dist-test-slave-d59m krb5kdc[13562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577191, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:11.403481 13650 init.cc:377] Logged in from keytab as kudu/127.22.225.189@KRBTEST.COM (short username kudu)
I20260928 06:33:11.404023 13650 webserver.cc:460] Webserver started at http://127.22.225.189:44901/ using document root <none> and password file <none>
I20260928 06:33:11.404295 13650 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:11.404342 13650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:11.404491 13650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:11.405640 13650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/minicluster-data/master-1/data/instance:
uuid: "57be79ee1ec248d8845319b34f696031"
format_stamp: "Formatted at 2026-09-28 06:33:11 on dist-test-slave-d59m"
server_key: "3105d3c8aa62cece19ed6b59fa0f9009"
server_key_iv: "4dacecb9337cc8ec601934300827167c"
server_key_version: "encryptionkey@0"
I20260928 06:33:11.406028 13650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/minicluster-data/master-1/wal/instance:
uuid: "57be79ee1ec248d8845319b34f696031"
format_stamp: "Formatted at 2026-09-28 06:33:11 on dist-test-slave-d59m"
server_key: "3105d3c8aa62cece19ed6b59fa0f9009"
server_key_iv: "4dacecb9337cc8ec601934300827167c"
server_key_version: "encryptionkey@0"
I20260928 06:33:11.407370 13650 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:33:11.408151 13665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:11.408319 13650 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:33:11.408411 13650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/minicluster-data/master-1/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/minicluster-data/master-1/wal
uuid: "57be79ee1ec248d8845319b34f696031"
format_stamp: "Formatted at 2026-09-28 06:33:11 on dist-test-slave-d59m"
server_key: "3105d3c8aa62cece19ed6b59fa0f9009"
server_key_iv: "4dacecb9337cc8ec601934300827167c"
server_key_version: "encryptionkey@0"
I20260928 06:33:11.408492 13650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:11.421649 13650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:11.423962 13650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:11.424151 13650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:11.428534 13650 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.189:37899
I20260928 06:33:11.428592 13717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.189:37899 every 8 connection(s)
I20260928 06:33:11.429011 13650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/minicluster-data/master-1/data/info.pb
I20260928 06:33:11.429993 13718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:11.431664 13718 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.225.190" port: 46789 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:33:11 dist-test-slave-d59m krb5kdc[13562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577191, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.189@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:33:11.438097 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 13650
I20260928 06:33:11.438192 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/minicluster-data/master-1/wal/instance
I20260928 06:33:11.438479 23430 external_mini_cluster.cc:1428] Setting key 1b2ff9e28048e4e433c74173d025ba23
I20260928 06:33:11.441159 13646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:11.432406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54924 (local address 127.22.225.190:46789)
0928 06:33:11.432499 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:11.432501 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:11.432707 (+ 206us) server_negotiation.cc:407] Connection header received
0928 06:33:11.433183 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:11.433184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:11.433198 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:11.433261 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:11.433878 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:11.434374 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:11.435385 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:11.435668 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:11.435728 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:11.437951 (+ 2223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:11.437953 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:11.437953 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:11.437960 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:11.439652 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:11.440090 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:11.440092 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:11.440093 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:11.440156 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:11.440426 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:11.440427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:11.440427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:11.440544 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:11.440688 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:11.440972 (+ 284us) server_negotiation.cc:299] Negotiation successful
0928 06:33:11.441061 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 06:33:11.441373 13720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:11.432384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:46789 (local address 127.0.0.1:54924)
0928 06:33:11.432564 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:11.432627 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:11.433093 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:11.433314 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:11.433317 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:11.433563 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:11.433765 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:11.433770 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:11.434470 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:11.434470 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:11.435307 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:11.435310 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:11.435750 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:11.435750 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:11.435864 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:11.436377 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:11.436385 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:11.437835 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:11.439741 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:11.439748 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:11.439750 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:11.439979 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:11.440233 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:11.440234 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:11.440234 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:11.440349 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:11.440680 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:11.440687 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:11.440835 (+ 148us) client_negotiation.cc:769] Sending connection context
0928 06:33:11.440952 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 06:33:11.441128 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":92,"threads_started":1}
I20260928 06:33:11.442015 13718 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.225.189" port: 37899 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.22.225.188@KRBTEST.COM; defaulting to no policy
Sep 28 06:33:11 dist-test-slave-d59m krb5kdc[13562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577191, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.189@KRBTEST.COM for kudu/127.22.225.189@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.225.188@KRBTEST.COM" created.
I20260928 06:33:11.449867 13720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:11.442306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.189:37899 (local address 127.0.0.1:48758)
0928 06:33:11.442386 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:11.442396 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:11.442443 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:11.442712 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:11.442713 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:11.442857 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:11.442904 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:11.442906 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:11.443752 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:11.443753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:11.444263 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:11.444264 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:11.444823 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:11.444824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:11.444889 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:11.445328 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:11.445333 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:11.446638 (+ 1305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:11.448544 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:11.448545 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:11.448546 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:11.448805 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:11.449059 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:11.449061 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:11.449061 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:11.449129 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:11.449486 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:11.449487 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:11.449537 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 06:33:11.449620 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 06:33:11.449729 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260928 06:33:11.450335 13722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:11.442380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48758 (local address 127.22.225.189:37899)
0928 06:33:11.442548 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:11.442558 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:11.442575 (+ 17us) server_negotiation.cc:407] Connection header received
0928 06:33:11.442617 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:11.442618 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:11.442630 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:11.442712 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:11.442981 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:11.443668 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:11.444355 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:11.444736 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:11.444825 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:11.446732 (+ 1907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:11.446739 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:11.446739 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:11.446746 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:11.448437 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:11.448912 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:11.448916 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:11.448916 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:11.448983 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:11.449208 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:11.449210 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:11.449210 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:11.449407 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:11.449496 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:11.449680 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 06:33:11.449803 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":141,"thread_start_us":90,"threads_started":1}
I20260928 06:33:11.450989 13718 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.225.188" port: 34885 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:33:11.451376 13720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:11.451233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.188:34885 (local address 127.0.0.1:43874)
0928 06:33:11.451286 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:11.451341 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.188:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 06:33:11.451501 13666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.225.188:34885: connect: Connection refused (error 111)
I20260928 06:33:11.451967 13720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:11.451844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.188:34885 (local address 127.0.0.1:43886)
0928 06:33:11.451901 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:11.451935 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.188:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260928 06:33:11.452159 13718 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.225.188:34885: Network error: Client connection negotiation failed: client connection to 127.22.225.188:34885: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.225.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.188.keytab.
Entry for principal kudu/127.22.225.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.188.keytab.
I20260928 06:33:11.463728 23430 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcCVV2M/build/release/bin/kudu
/tmp/dist-test-taskcCVV2M/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.188:34885
--webserver_interface=127.22.225.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.188.keytab
--principal=kudu/127.22.225.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.225.148:39585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.225.190:46789,127.22.225.189:37899,127.22.225.188:34885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:33:11.482551 13643 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.225.189" port: 37899 } attempt: 3
Sep 28 06:33:11 dist-test-slave-d59m krb5kdc[13562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577191, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.190@KRBTEST.COM for kudu/127.22.225.189@KRBTEST.COM
I20260928 06:33:11.491150 13645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:11.482905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.189:37899 (local address 127.0.0.1:48762)
0928 06:33:11.482969 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:11.482982 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:11.483027 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:11.483229 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:11.483230 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:11.483433 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:11.483509 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:11.483511 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:11.484504 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:11.484506 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:11.485411 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:11.485414 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:11.485923 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:11.485924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:11.486003 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:11.486526 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:11.486531 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:11.488105 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:11.489968 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:11.489970 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:11.489970 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:11.490199 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:11.490434 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:11.490434 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:11.490435 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:11.490512 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:11.490818 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:11.490819 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:11.490870 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 06:33:11.490943 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 06:33:11.491051 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 06:33:11.491176 13722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:11.482992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48762 (local address 127.22.225.189:37899)
0928 06:33:11.483062 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:11.483065 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:11.483081 (+ 16us) server_negotiation.cc:407] Connection header received
0928 06:33:11.483113 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:11.483114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:11.483127 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:11.483209 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:11.483596 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:11.484393 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:11.485504 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:11.485838 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:11.485915 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:11.488198 (+ 2283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:11.488200 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:11.488200 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:11.488208 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:11.489877 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:11.490288 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:11.490289 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:11.490290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:11.490354 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:11.490593 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:11.490594 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:11.490595 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:11.490745 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:11.490820 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:11.490986 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 06:33:11.491071 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 06:33:11.491840 13643 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.225.188" port: 34885 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:33:11.492199 13645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:11.492075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.188:34885 (local address 127.0.0.1:43890)
0928 06:33:11.492135 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:11.492172 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.188:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 06:33:11.492599 13645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:11.492471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.188:34885 (local address 127.0.0.1:43898)
0928 06:33:11.492523 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:11.492552 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.188:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 06:33:11.492758 13643 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.225.188:34885: Network error: Client connection negotiation failed: client connection to 127.22.225.188:34885: connect: Connection refused (error 111)
I20260928 06:33:11.501408 13718 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.225.188" port: 34885 } attempt: 1
I20260928 06:33:11.501904 13720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:11.501753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.188:34885 (local address 127.0.0.1:43910)
0928 06:33:11.501821 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:11.501866 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.188:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 06:33:11.502328 13720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:11.502224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.188:34885 (local address 127.0.0.1:43916)
0928 06:33:11.502274 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:11.502304 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.188:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 06:33:11.502458 13718 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.225.188:34885: Network error: Client connection negotiation failed: client connection to 127.22.225.188:34885: connect: Connection refused (error 111)
I20260928 06:33:11.523916 13643 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.225.188" port: 34885 } attempt: 1
I20260928 06:33:11.524452 13645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:11.524288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.188:34885 (local address 127.0.0.1:43924)
0928 06:33:11.524362 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:11.524413 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.188:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 06:33:11.524933 13645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:11.524742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.188:34885 (local address 127.0.0.1:43926)
0928 06:33:11.524877 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:11.524900 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.188:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":93,"threads_started":1}
W20260928 06:33:11.525036 13643 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.225.188:34885: Network error: Client connection negotiation failed: client connection to 127.22.225.188:34885: connect: Connection refused (error 111)
W20260928 06:33:11.552760 13725 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 06:33:11.552980 13725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:11.553000 13725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:11.553023 13725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:11.554251 13725 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:33:11.554297 13725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:11.554312 13725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:11.554325 13725 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:33:11.554337 13725 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:33:11.555866 13725 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.148:39585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.22.225.190:46789,127.22.225.189:37899,127.22.225.188:34885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/krb5kdc/kudu_127.22.225.188.keytab
--principal=kudu/127.22.225.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.188:34885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.22.225.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:33:11.556142 13725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:11.556403 13725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:11.561487 13734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:11.561488 13732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:11.561661 13731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:11.562337 13725 server_base.cc:1034] running on GCE node
I20260928 06:33:11.562641 13725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:11.562973 13725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:11.564150 13725 hybrid_clock.cc:648] HybridClock initialized: now 1790577191564127 us; error 39 us; skew 500 ppm
Sep 28 06:33:11 dist-test-slave-d59m krb5kdc[13562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577191, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:11.566830 13725 init.cc:377] Logged in from keytab as kudu/127.22.225.188@KRBTEST.COM (short username kudu)
I20260928 06:33:11.567318 13725 webserver.cc:460] Webserver started at http://127.22.225.188:46767/ using document root <none> and password file <none>
I20260928 06:33:11.567553 13725 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:11.567621 13725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:11.567776 13725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:11.568961 13725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/minicluster-data/master-2/data/instance:
uuid: "f5103cadac8e4f61b924dda3a102d319"
format_stamp: "Formatted at 2026-09-28 06:33:11 on dist-test-slave-d59m"
server_key: "942463d0b33250bcebf7afb79a864de2"
server_key_iv: "8d7c5b070073ed4990afc87ee64283f2"
server_key_version: "encryptionkey@0"
I20260928 06:33:11.569326 13725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/minicluster-data/master-2/wal/instance:
uuid: "f5103cadac8e4f61b924dda3a102d319"
format_stamp: "Formatted at 2026-09-28 06:33:11 on dist-test-slave-d59m"
server_key: "942463d0b33250bcebf7afb79a864de2"
server_key_iv: "8d7c5b070073ed4990afc87ee64283f2"
server_key_version: "encryptionkey@0"
I20260928 06:33:11.570634 13718 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.225.188" port: 34885 } attempt: 2
I20260928 06:33:11.570778 13725 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:11.571069 13720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:11.570945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.188:34885 (local address 127.0.0.1:43928)
0928 06:33:11.571002 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:11.571034 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.188:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 06:33:11.571434 13720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:11.571335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.188:34885 (local address 127.0.0.1:43936)
0928 06:33:11.571387 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:11.571415 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.188:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260928 06:33:11.571521 13718 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.225.188:34885: Network error: Client connection negotiation failed: client connection to 127.22.225.188:34885: connect: Connection refused (error 111)
I20260928 06:33:11.571678 13741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:11.571834 13725 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:33:11.571949 13725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/minicluster-data/master-2/data,/tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/minicluster-data/master-2/wal
uuid: "f5103cadac8e4f61b924dda3a102d319"
format_stamp: "Formatted at 2026-09-28 06:33:11 on dist-test-slave-d59m"
server_key: "942463d0b33250bcebf7afb79a864de2"
server_key_iv: "8d7c5b070073ed4990afc87ee64283f2"
server_key_version: "encryptionkey@0"
I20260928 06:33:11.572048 13725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:11.585765 13725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:11.588294 13725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:11.588469 13725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:11.593271 13725 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.188:34885
I20260928 06:33:11.593298 13793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.188:34885 every 8 connection(s)
I20260928 06:33:11.593714 13725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/minicluster-data/master-2/data/info.pb
I20260928 06:33:11.594648 13794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:11.596225 13794 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.225.190" port: 46789 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:33:11.599022 23430 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcCVV2M/build/release/bin/kudu as pid 13725
I20260928 06:33:11.599141 23430 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcCVV2M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576899143097-23430-0/minicluster-data/master-2/wal/instance
I20260928 06:33:11.599355 23430 external_mini_cluster.cc:1428] Setting key be0e49fa99187a96c1dd859db0ac67c8
I20260928 06:33:11.600167 13643 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.225.188" port: 34885 } attempt: 2
Sep 28 06:33:11 dist-test-slave-d59m krb5kdc[13562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577191, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.188@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
Sep 28 06:33:11 dist-test-slave-d59m krb5kdc[13562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.225.188@KRBTEST.COM
Sep 28 06:33:11 dist-test-slave-d59m krb5kdc[13562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577191, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.190@KRBTEST.COM for kudu/127.22.225.188@KRBTEST.COM
I20260928 06:33:11.607695 13646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:11.597154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54930 (local address 127.22.225.190:46789)
0928 06:33:11.597229 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:11.597232 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:11.597493 (+ 261us) server_negotiation.cc:407] Connection header received
0928 06:33:11.598116 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:11.598117 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:11.598127 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:11.598184 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:11.598992 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:11.599737 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:11.601191 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:11.601478 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:11.601523 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:11.604334 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:11.604335 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:11.604336 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:11.604343 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:11.606042 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:11.606614 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:11.606615 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:11.606616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:11.606677 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:11.606970 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:11.606971 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:11.606971 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:11.607100 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:11.607163 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:11.607514 (+ 351us) server_negotiation.cc:299] Negotiation successful
0928 06:33:11.607609 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 06:33:11.608220 13796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:11.597139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.190:46789 (local address 127.0.0.1:54930)
0928 06:33:11.597339 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:11.597391 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:11.598000 (+ 609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:11.598255 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:11.598259 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:11.598609 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:11.598888 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:11.598894 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:11.599827 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:11.599829 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:11.601084 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:11.601087 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:11.601623 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:11.601624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:11.601756 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:11.602366 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:11.602376 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:11.604269 (+ 1893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:11.606145 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:11.606152 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:11.606154 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:11.606542 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:11.606759 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:11.606760 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:11.606761 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:11.606899 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:11.607194 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:11.607201 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:11.607361 (+ 160us) client_negotiation.cc:769] Sending connection context
0928 06:33:11.607828 (+ 467us) client_negotiation.cc:240] Negotiation successful
0928 06:33:11.607994 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":101,"threads_started":1}
I20260928 06:33:11.608812 13794 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.225.189" port: 37899 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:33:11.611359 13798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:11.599860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43952 (local address 127.22.225.188:34885)
0928 06:33:11.600053 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:11.600058 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:11.600738 (+ 680us) server_negotiation.cc:407] Connection header received
0928 06:33:11.601107 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:11.601108 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:11.601119 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:11.601182 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:11.601944 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:11.602509 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:11.603417 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:11.603841 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:11.603904 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:11.607523 (+ 3619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:11.607524 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:11.607525 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:11.607531 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:11.609919 (+ 2388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:11.610393 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:11.610397 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:11.610397 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:11.610457 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:11.610701 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:11.610702 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:11.610702 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:11.610883 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:11.611130 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:11.611173 (+ 43us) server_negotiation.cc:299] Negotiation successful
0928 06:33:11.611250 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":108,"threads_started":1}
I20260928 06:33:11.612689 13645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:11.600410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.188:34885 (local address 127.0.0.1:43964)
0928 06:33:11.600455 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:11.600466 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:11.600500 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:11.600801 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:11.600802 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:11.600971 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:11.601009 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:11.601011 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:11.602580 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:11.602582 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:11.603356 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:11.603359 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:11.603993 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:11.603994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:11.604065 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:11.604949 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:11.604955 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:11.607312 (+ 2357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:11.611531 (+ 4219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:11.611532 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:11.611532 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:11.611740 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:11.611957 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:11.611958 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:11.611958 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:11.612021 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:11.612319 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:11.612320 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:11.612368 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 06:33:11.612417 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 06:33:11.612531 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 06:33:11.613309 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:11.600461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43964 (local address 127.22.225.188:34885)
0928 06:33:11.600602 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:11.600604 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:11.600632 (+ 28us) server_negotiation.cc:407] Connection header received
0928 06:33:11.600655 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:11.600657 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:11.600678 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:11.600788 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:11.601091 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:11.602472 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:11.603485 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:11.603901 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:11.604635 (+ 734us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:11.607411 (+ 2776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:11.607420 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:11.607421 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:11.607429 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:11.610988 (+ 3559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:11.611827 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:11.611827 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:11.611828 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:11.611884 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:11.612100 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:11.612101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:11.612102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:11.612234 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:11.613021 (+ 787us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:11.613064 (+ 43us) server_negotiation.cc:299] Negotiation successful
0928 06:33:11.613147 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":70,"threads_started":1}
Sep 28 06:33:11 dist-test-slave-d59m krb5kdc[13562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577191, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.188@KRBTEST.COM for kudu/127.22.225.189@KRBTEST.COM
I20260928 06:33:11.616133 13643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d367f41741664aefbf4679dec7413564: Bootstrap starting.
I20260928 06:33:11.617002 13643 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d367f41741664aefbf4679dec7413564: Neither blocks nor log segments found. Creating new log.
Sep 28 06:33:11 dist-test-slave-d59m krb5kdc[13562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.225.190@KRBTEST.COM
I20260928 06:33:11.617328 13643 log.cc:824] T 00000000000000000000000000000000 P d367f41741664aefbf4679dec7413564: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:11.618337 13643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d367f41741664aefbf4679dec7413564: No bootstrap required, opened a new log
I20260928 06:33:11.619318 13722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:11.609170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48766 (local address 127.22.225.189:37899)
0928 06:33:11.609226 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:11.609228 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:11.609266 (+ 38us) server_negotiation.cc:407] Connection header received
0928 06:33:11.609357 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:11.609358 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:11.609369 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:11.609397 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:11.609854 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:11.610433 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:11.611250 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:11.611493 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:11.611544 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:11.614081 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:11.614083 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:11.614083 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:11.614091 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:11.617921 (+ 3830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:11.618343 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:11.618345 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:11.618345 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:11.618407 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:11.618695 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:11.618697 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:11.618697 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:11.618843 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:11.618887 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:11.619116 (+ 229us) server_negotiation.cc:299] Negotiation successful
0928 06:33:11.619193 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 06:33:11.619841 13796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:11.609071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.189:37899 (local address 127.0.0.1:48766)
0928 06:33:11.609217 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:11.609228 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:11.609289 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:11.609457 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:11.609458 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:11.609719 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:11.609757 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:11.609759 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:11.610560 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:11.610560 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:11.611151 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:11.611154 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:11.611716 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:11.611717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:11.611770 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:11.612449 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:11.612455 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:11.613980 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:11.618030 (+ 4050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:11.618031 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:11.618031 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:11.618249 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:11.618541 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:11.618541 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:11.618542 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:11.618611 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:11.618939 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:11.618939 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:11.618987 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 06:33:11.619624 (+ 637us) client_negotiation.cc:240] Negotiation successful
0928 06:33:11.619727 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130}
I20260928 06:33:11.620406 13794 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.225.188" port: 34885 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:33:11.620776 13643 raft_consensus.cc:348] T 00000000000000000000000000000000 P d367f41741664aefbf4679dec7413564 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d367f41741664aefbf4679dec7413564" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 46789 } } peers { permanent_uuid: "57be79ee1ec248d8845319b34f696031" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 37899 } } peers { permanent_uuid: "f5103cadac8e4f61b924dda3a102d319" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 34885 } }
I20260928 06:33:11.620923 13643 raft_consensus.cc:729] T 00000000000000000000000000000000 P d367f41741664aefbf4679dec7413564 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d367f41741664aefbf4679dec7413564, State: Initialized, Role: FOLLOWER
I20260928 06:33:11.620997 13643 consensus_queue.cc:260] T 00000000000000000000000000000000 P d367f41741664aefbf4679dec7413564 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d367f41741664aefbf4679dec7413564" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 46789 } } peers { permanent_uuid: "57be79ee1ec248d8845319b34f696031" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 37899 } } peers { permanent_uuid: "f5103cadac8e4f61b924dda3a102d319" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 34885 } }
I20260928 06:33:11.621321 13643 sys_catalog.cc:564] T 00000000000000000000000000000000 P d367f41741664aefbf4679dec7413564 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:33:11.621521 13646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:11.612483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54932 (local address 127.22.225.190:46789)
0928 06:33:11.612574 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:11.612576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:11.612677 (+ 101us) server_negotiation.cc:407] Connection header received
0928 06:33:11.612767 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:11.612768 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:11.612778 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:11.612809 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:11.613456 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:11.613983 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:11.614833 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:11.615048 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:11.615131 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:11.618481 (+ 3350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:11.618483 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:11.618483 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:11.618490 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:11.620191 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:11.620568 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:11.620569 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:11.620570 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:11.620662 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:11.620876 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:11.620877 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:11.620877 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:11.621006 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:11.621128 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:11.621352 (+ 224us) server_negotiation.cc:299] Negotiation successful
0928 06:33:11.621429 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
Sep 28 06:33:11 dist-test-slave-d59m krb5kdc[13562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577191, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.188@KRBTEST.COM for kudu/127.22.225.188@KRBTEST.COM
I20260928 06:33:11.623802 13575 master_runner.cc:186] Error getting master registration for 127.22.225.189:37899: OK, instance_id { permanent_uuid: "57be79ee1ec248d8845319b34f696031" instance_seqno: 1790577191427255 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 06:33:11.626564 13575 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 28 06:33:11 dist-test-slave-d59m krb5kdc[13562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.225.189@KRBTEST.COM
I20260928 06:33:11.628666 13801 sys_catalog.cc:455] T 00000000000000000000000000000000 P d367f41741664aefbf4679dec7413564 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d367f41741664aefbf4679dec7413564" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 46789 } } peers { permanent_uuid: "57be79ee1ec248d8845319b34f696031" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 37899 } } peers { permanent_uuid: "f5103cadac8e4f61b924dda3a102d319" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 34885 } } }
I20260928 06:33:11.628773 13801 sys_catalog.cc:458] T 00000000000000000000000000000000 P d367f41741664aefbf4679dec7413564 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:33:11.628890 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:11.620743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43976 (local address 127.22.225.188:34885)
0928 06:33:11.620779 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:11.620781 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:11.620819 (+ 38us) server_negotiation.cc:407] Connection header received
0928 06:33:11.620878 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:11.620879 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:11.620891 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:11.620942 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:11.621461 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:11.622173 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:11.623422 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:11.623730 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:11.623765 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:11.625964 (+ 2199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:11.625966 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:11.625966 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:11.625972 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:11.627652 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:11.628038 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:11.628040 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:11.628040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:11.628098 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:11.628298 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:11.628299 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:11.628300 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:11.628485 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:11.628559 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:11.628734 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 06:33:11.628808 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 06:33:11.629359 13796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:11.620711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.188:34885 (local address 127.0.0.1:43976)
0928 06:33:11.620777 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:11.620787 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:11.620831 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:11.621070 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:11.621071 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:11.621304 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:11.621340 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:11.621342 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:11.622559 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:11.622560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:11.623333 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:11.623335 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:11.623822 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:11.623823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:11.623863 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:11.624287 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:11.624293 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:11.625873 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:11.627745 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:11.627746 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:11.627747 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:11.627953 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:11.628179 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:11.628179 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:11.628180 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:11.628233 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:11.628605 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:11.628605 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:11.628641 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 06:33:11.629192 (+ 551us) client_negotiation.cc:240] Negotiation successful
0928 06:33:11.629282 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 06:33:11.629724 13812 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d367f41741664aefbf4679dec7413564: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:33:11.629777 13812 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:33:11.631045 13794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5103cadac8e4f61b924dda3a102d319: Bootstrap starting.
I20260928 06:33:11.631750 13794 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5103cadac8e4f61b924dda3a102d319: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:11.631783 13722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:11.622390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48782 (local address 127.22.225.189:37899)
0928 06:33:11.622424 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:11.622425 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:11.622495 (+ 70us) server_negotiation.cc:407] Connection header received
0928 06:33:11.622592 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:11.622593 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:11.622602 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:11.622630 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:11.623242 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:11.623917 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:11.624958 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:11.625312 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:11.625356 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:11.628566 (+ 3210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:11.628568 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:11.628568 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:11.628612 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:11.630251 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:11.630676 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:11.630677 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:11.630677 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:11.630734 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:11.630925 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:11.630926 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:11.630926 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:11.631059 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:11.631218 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:11.631622 (+ 404us) server_negotiation.cc:299] Negotiation successful
0928 06:33:11.631698 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 06:33:11.632037 13794 log.cc:824] T 00000000000000000000000000000000 P f5103cadac8e4f61b924dda3a102d319: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:11.633159 23430 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:33:11.633200 13794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5103cadac8e4f61b924dda3a102d319: No bootstrap required, opened a new log
I20260928 06:33:11.634028 13617 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d367f41741664aefbf4679dec7413564"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54932
I20260928 06:33:11.634150 13617 raft_consensus.cc:482] T 00000000000000000000000000000000 P d367f41741664aefbf4679dec7413564 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 06:33:11.634195 13617 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d367f41741664aefbf4679dec7413564 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:11.635064 13617 raft_consensus.cc:504] T 00000000000000000000000000000000 P d367f41741664aefbf4679dec7413564 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d367f41741664aefbf4679dec7413564" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 46789 } } peers { permanent_uuid: "57be79ee1ec248d8845319b34f696031" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 37899 } } peers { permanent_uuid: "f5103cadac8e4f61b924dda3a102d319" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 34885 } }
I20260928 06:33:11.635083 13794 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5103cadac8e4f61b924dda3a102d319 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d367f41741664aefbf4679dec7413564" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 46789 } } peers { permanent_uuid: "57be79ee1ec248d8845319b34f696031" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 37899 } } peers { permanent_uuid: "f5103cadac8e4f61b924dda3a102d319" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 34885 } }
I20260928 06:33:11.635216 13794 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5103cadac8e4f61b924dda3a102d319 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5103cadac8e4f61b924dda3a102d319, State: Initialized, Role: FOLLOWER
I20260928 06:33:11.635288 13794 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5103cadac8e4f61b924dda3a102d319 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d367f41741664aefbf4679dec7413564" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 46789 } } peers { permanent_uuid: "57be79ee1ec248d8845319b34f696031" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 37899 } } peers { permanent_uuid: "f5103cadac8e4f61b924dda3a102d319" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 34885 } }
I20260928 06:33:11.635310 13617 leader_election.cc:288] T 00000000000000000000000000000000 P d367f41741664aefbf4679dec7413564 [CANDIDATE]: Term 1 election: Requested vote from peers 57be79ee1ec248d8845319b34f696031 (127.22.225.189:37899), f5103cadac8e4f61b924dda3a102d319 (127.22.225.188:34885)
I20260928 06:33:11.635584 13794 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5103cadac8e4f61b924dda3a102d319 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:33:11.636092 13768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d367f41741664aefbf4679dec7413564" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f5103cadac8e4f61b924dda3a102d319"
I20260928 06:33:11.636231 13768 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5103cadac8e4f61b924dda3a102d319 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:11.637173 13768 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f5103cadac8e4f61b924dda3a102d319 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d367f41741664aefbf4679dec7413564 in term 1.
I20260928 06:33:11.637457 13591 leader_election.cc:302] T 00000000000000000000000000000000 P d367f41741664aefbf4679dec7413564 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d367f41741664aefbf4679dec7413564, f5103cadac8e4f61b924dda3a102d319; no voters:
I20260928 06:33:11.637594 13801 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d367f41741664aefbf4679dec7413564 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:33:11.637755 13801 raft_consensus.cc:686] T 00000000000000000000000000000000 P d367f41741664aefbf4679dec7413564 [term 1 LEADER]: Becoming Leader. State: Replica: d367f41741664aefbf4679dec7413564, State: Running, Role: LEADER
I20260928 06:33:11.637844 13801 consensus_queue.cc:237] T 00000000000000000000000000000000 P d367f41741664aefbf4679dec7413564 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d367f41741664aefbf4679dec7413564" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 46789 } } peers { permanent_uuid: "57be79ee1ec248d8845319b34f696031" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 37899 } } peers { permanent_uuid: "f5103cadac8e4f61b924dda3a102d319" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 34885 } }
W20260928 06:33:11.637898 13830 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f5103cadac8e4f61b924dda3a102d319: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:33:11.637941 13830 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:33:11.638339 13832 sys_catalog.cc:455] T 00000000000000000000000000000000 P d367f41741664aefbf4679dec7413564 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d367f41741664aefbf4679dec7413564. Latest consensus state: current_term: 1 leader_uuid: "d367f41741664aefbf4679dec7413564" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d367f41741664aefbf4679dec7413564" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 46789 } } peers { permanent_uuid: "57be79ee1ec248d8845319b34f696031" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 37899 } } peers { permanent_uuid: "f5103cadac8e4f61b924dda3a102d319" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 34885 } } }
I20260928 06:33:11.638535 13832 sys_catalog.cc:458] T 00000000000000000000000000000000 P d367f41741664aefbf4679dec7413564 [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:11.638813 13834 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:33:11.639027 13834 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:33:11.640482 13768 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f5103cadac8e4f61b924dda3a102d319 [term 1 FOLLOWER]: Refusing update from remote peer d367f41741664aefbf4679dec7413564: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:33:11.640820 13832 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d367f41741664aefbf4679dec7413564 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5103cadac8e4f61b924dda3a102d319" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 34885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:33:11.639683 13725 master_runner.cc:186] Error getting master registration for 127.22.225.189:37899: OK, instance_id { permanent_uuid: "57be79ee1ec248d8845319b34f696031" instance_seqno: 1790577191427255 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 06:33:11.642424 13725 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 06:33:11.642978 13836 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5103cadac8e4f61b924dda3a102d319 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d367f41741664aefbf4679dec7413564" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d367f41741664aefbf4679dec7413564" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 46789 } } peers { permanent_uuid: "57be79ee1ec248d8845319b34f696031" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 37899 } } peers { permanent_uuid: "f5103cadac8e4f61b924dda3a102d319" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 34885 } } }
I20260928 06:33:11.643113 13836 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5103cadac8e4f61b924dda3a102d319 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:33:11.643198 13836 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5103cadac8e4f61b924dda3a102d319 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d367f41741664aefbf4679dec7413564. Latest consensus state: current_term: 1 leader_uuid: "d367f41741664aefbf4679dec7413564" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d367f41741664aefbf4679dec7413564" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 46789 } } peers { permanent_uuid: "57be79ee1ec248d8845319b34f696031" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 37899 } } peers { permanent_uuid: "f5103cadac8e4f61b924dda3a102d319" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 34885 } } }
I20260928 06:33:11.643281 13836 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5103cadac8e4f61b924dda3a102d319 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:33:11.644163 13801 sys_catalog.cc:455] T 00000000000000000000000000000000 P d367f41741664aefbf4679dec7413564 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d367f41741664aefbf4679dec7413564" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d367f41741664aefbf4679dec7413564" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 46789 } } peers { permanent_uuid: "57be79ee1ec248d8845319b34f696031" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 37899 } } peers { permanent_uuid: "f5103cadac8e4f61b924dda3a102d319" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 34885 } } }
I20260928 06:33:11.644259 13801 sys_catalog.cc:458] T 00000000000000000000000000000000 P d367f41741664aefbf4679dec7413564 [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:11.644447 13835 mvcc.cc:205] Tried to move back new op lower bound from 7334204176957186048 to 7334204176949305344. Current Snapshot: MvccSnapshot[applied={T|T < 7334204176957186048}]
I20260928 06:33:11.644611 13836 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5103cadac8e4f61b924dda3a102d319 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d367f41741664aefbf4679dec7413564" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d367f41741664aefbf4679dec7413564" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 46789 } } peers { permanent_uuid: "57be79ee1ec248d8845319b34f696031" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 37899 } } peers { permanent_uuid: "f5103cadac8e4f61b924dda3a102d319" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 34885 } } }
I20260928 06:33:11.644683 13836 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5103cadac8e4f61b924dda3a102d319 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:33:11.645135 13834 catalog_manager.cc:1347] Generated new cluster ID: a5f0f723a62b4d25a3ba471f73f0997f
I20260928 06:33:11.645186 13834 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:33:11.658149 13646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:11.649260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54936 (local address 127.22.225.190:46789)
0928 06:33:11.649313 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:11.649315 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:11.649950 (+ 635us) server_negotiation.cc:407] Connection header received
0928 06:33:11.650033 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:11.650035 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:11.650060 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:11.650108 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:11.650930 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:11.651583 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:11.652708 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:11.652974 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:11.653041 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:11.654757 (+ 1716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:11.654759 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:11.654759 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:11.654768 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:11.656453 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:11.656873 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:11.656874 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:11.656875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:11.656940 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:11.657132 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:11.657133 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:11.657133 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:11.657257 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:11.657354 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:11.657968 (+ 614us) server_negotiation.cc:299] Negotiation successful
0928 06:33:11.658049 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":19}
I20260928 06:33:11.662635 13718 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.225.188" port: 34885 } attempt: 3
Sep 28 06:33:11 dist-test-slave-d59m krb5kdc[13562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577191, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.225.189@KRBTEST.COM for kudu/127.22.225.188@KRBTEST.COM
I20260928 06:33:11.673152 13798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:11.663035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44000 (local address 127.22.225.188:34885)
0928 06:33:11.663077 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:11.663078 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:11.663086 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:33:11.663107 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:11.663108 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:11.663117 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:11.663180 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:11.663493 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:11.664195 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:11.665179 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:11.665499 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:11.665557 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:11.667658 (+ 2101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:11.667660 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:11.667660 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:11.667667 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:11.672015 (+ 4348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:11.672396 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:11.672397 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:11.672398 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:11.672447 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:11.672640 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:11.672641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:11.672641 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:11.672760 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:11.672814 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:11.672973 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 06:33:11.673047 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 06:33:11.673161 13720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:11.662941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.188:34885 (local address 127.0.0.1:44000)
0928 06:33:11.662982 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:11.662993 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:11.663033 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:11.663198 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:11.663199 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:11.663380 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:11.663425 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:11.663427 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:11.664265 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:11.664266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:11.665104 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:11.665106 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:11.665575 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:11.665576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:11.665633 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:11.666028 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:11.666033 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:11.667581 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:11.672078 (+ 4497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:11.672087 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:11.672087 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:11.672316 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:11.672507 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:11.672508 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:11.672508 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:11.672587 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:11.672818 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:11.672818 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:11.672867 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 06:33:11.672940 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 06:33:11.673038 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260928 06:33:11.674126 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:11.649484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43988 (local address 127.22.225.188:34885)
0928 06:33:11.649524 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:11.649526 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:11.650872 (+ 1346us) server_negotiation.cc:407] Connection header received
0928 06:33:11.651845 (+ 973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:11.651846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:11.651858 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:11.651889 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:11.655703 (+ 3814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:11.656343 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:11.660943 (+ 4600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:11.661236 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:11.661283 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:11.668526 (+ 7243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:11.668527 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:11.668528 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:11.668535 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:11.671432 (+ 2897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:11.672513 (+ 1081us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:11.672513 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:11.672514 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:11.672548 (+ 34us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:11.673340 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:11.673341 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:11.673342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:11.673459 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:11.673550 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:11.673959 (+ 409us) server_negotiation.cc:299] Negotiation successful
0928 06:33:11.674039 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:33:11.675169 13718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57be79ee1ec248d8845319b34f696031: Bootstrap starting.
I20260928 06:33:11.675850 13834 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:33:11.675976 13718 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57be79ee1ec248d8845319b34f696031: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:11.676307 13718 log.cc:824] T 00000000000000000000000000000000 P 57be79ee1ec248d8845319b34f696031: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:11.676805 13722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:11.650354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48788 (local address 127.22.225.189:37899)
0928 06:33:11.650401 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:11.650403 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:11.652270 (+ 1867us) server_negotiation.cc:407] Connection header received
0928 06:33:11.652569 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:11.652599 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:11.652614 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:11.652648 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:11.656965 (+ 4317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:11.657622 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:11.661505 (+ 3883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:11.661838 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:11.661882 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:11.668848 (+ 6966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:11.668849 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:11.668849 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:11.668856 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:11.670618 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:11.675672 (+ 5054us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:11.675673 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:11.675674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:11.675755 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:11.676069 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:11.676070 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:11.676070 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:11.676204 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:11.676292 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:11.676621 (+ 329us) server_negotiation.cc:299] Negotiation successful
0928 06:33:11.676710 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 06:33:11.677395 13834 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:33:11.678160 13718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57be79ee1ec248d8845319b34f696031: No bootstrap required, opened a new log
I20260928 06:33:11.680194 13718 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57be79ee1ec248d8845319b34f696031 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d367f41741664aefbf4679dec7413564" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 46789 } } peers { permanent_uuid: "57be79ee1ec248d8845319b34f696031" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 37899 } } peers { permanent_uuid: "f5103cadac8e4f61b924dda3a102d319" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 34885 } }
I20260928 06:33:11.680341 13718 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57be79ee1ec248d8845319b34f696031 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57be79ee1ec248d8845319b34f696031, State: Initialized, Role: FOLLOWER
I20260928 06:33:11.680425 13718 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57be79ee1ec248d8845319b34f696031 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d367f41741664aefbf4679dec7413564" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 46789 } } peers { permanent_uuid: "57be79ee1ec248d8845319b34f696031" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 37899 } } peers { permanent_uuid: "f5103cadac8e4f61b924dda3a102d319" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 34885 } }
I20260928 06:33:11.680779 13718 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57be79ee1ec248d8845319b34f696031 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:33:11.680845 13692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d367f41741664aefbf4679dec7413564" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "57be79ee1ec248d8845319b34f696031"
I20260928 06:33:11.680959 13692 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57be79ee1ec248d8845319b34f696031 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:11.681804 13692 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 57be79ee1ec248d8845319b34f696031 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d367f41741664aefbf4679dec7413564 in term 1.
I20260928 06:33:11.682235 13691 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 57be79ee1ec248d8845319b34f696031 [term 1 FOLLOWER]: Refusing update from remote peer d367f41741664aefbf4679dec7413564: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:33:11.689311 13801 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d367f41741664aefbf4679dec7413564 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57be79ee1ec248d8845319b34f696031" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 37899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:33:11.693260 13846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57be79ee1ec248d8845319b34f696031 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d367f41741664aefbf4679dec7413564" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d367f41741664aefbf4679dec7413564" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 46789 } } peers { permanent_uuid: "57be79ee1ec248d8845319b34f696031" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 37899 } } peers { permanent_uuid: "f5103cadac8e4f61b924dda3a102d319" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 34885 } } }
I20260928 06:33:11.693367 13846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57be79ee1ec248d8845319b34f696031 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:33:11.693425 13846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57be79ee1ec248d8845319b34f696031 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d367f41741664aefbf4679dec7413564. Latest consensus state: current_term: 1 leader_uuid: "d367f41741664aefbf4679dec7413564" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d367f41741664aefbf4679dec7413564" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 46789 } } peers { permanent_uuid: "57be79ee1ec248d8845319b34f696031" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 37899 } } peers { permanent_uuid: "f5103cadac8e4f61b924dda3a102d319" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 34885 } } }
I20260928 06:33:11.693482 13846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57be79ee1ec248d8845319b34f696031 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:33:11.693516 13846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57be79ee1ec248d8845319b34f696031 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d367f41741664aefbf4679dec7413564" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d367f41741664aefbf4679dec7413564" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 46789 } } peers { permanent_uuid: "57be79ee1ec248d8845319b34f696031" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 37899 } } peers { permanent_uuid: "f5103cadac8e4f61b924dda3a102d319" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 34885 } } }
I20260928 06:33:11.693565 13846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57be79ee1ec248d8845319b34f696031 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 06:33:11.694376 13857 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 57be79ee1ec248d8845319b34f696031: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:33:11.694438 13857 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:33:11.701138 13859 mvcc.cc:205] Tried to move back new op lower bound from 7334204177098469376 to 7334204176949305344. Current Snapshot: MvccSnapshot[applied={T|T < 7334204176957186048}]
I20260928 06:33:11.702039 13832 sys_catalog.cc:455] T 00000000000000000000000000000000 P d367f41741664aefbf4679dec7413564 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d367f41741664aefbf4679dec7413564" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d367f41741664aefbf4679dec7413564" member_type: VOTER last_known_addr { host: "127.22.225.190" port: 46789 } } peers { permanent_uuid: "57be79ee1ec248d8845319b34f696031" member_type: VOTER last_known_addr { host: "127.22.225.189" port: 37899 } } peers { permanent_uuid: "f5103cadac8e4f61b924dda3a102d319" member_type: VOTER last_known_addr { host: "127.22.225.188" port: 34885 } } }
I20260928 06:33:11.702136 13832 sys_catalog.cc:458] T 00000000000000000000000000000000 P d367f41741664aefbf4679dec7413564 [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:11.704622 13834 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d367f41741664aefbf4679dec7413564: Generated new TSK 0
I20260928 06:33:11.704825 13834 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 06:33:11.753671 13799 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:33:11.742707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44014 (local address 127.22.225.188:34885)
0928 06:33:11.742754 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:11.742756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:11.752817 (+ 10061us) server_negotiation.cc:407] Connection header received
0928 06:33:11.752949 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:11.752951 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:11.752964 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:11.752999 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:11.753588 (+ 589us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44014: BlockingRecv error: recv got EOF from 127.0.0.1:44014 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 06:33:11.754460 13646 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:33:11.752947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54946 (local address 127.22.225.190:46789)
0928 06:33:11.752996 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:11.752998 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:11.753929 (+ 931us) server_negotiation.cc:407] Connection header received
0928 06:33:11.754032 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:11.754033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:11.754044 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:11.754074 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:11.754396 (+ 322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54946: BlockingRecv error: recv got EOF from 127.0.0.1:54946 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260928 06:33:11.755180 13722 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:33:11.753362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48792 (local address 127.22.225.189:37899)
0928 06:33:11.753406 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:11.753408 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:11.754663 (+ 1255us) server_negotiation.cc:407] Connection header received
0928 06:33:11.754767 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:11.754768 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:11.754781 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:11.754810 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:11.755100 (+ 290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48792: BlockingRecv error: recv got EOF from 127.0.0.1:48792 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 06:33:11.769371 13722 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:33:11.761965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48800 (local address 127.22.225.189:37899)
0928 06:33:11.762016 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:11.762018 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:11.768758 (+ 6740us) server_negotiation.cc:407] Connection header received
0928 06:33:11.768881 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:11.768882 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:11.768905 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:11.768934 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:11.769311 (+ 377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48800: BlockingRecv error: recv got EOF from 127.0.0.1:48800 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260928 06:33:11.778641 13722 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:33:11.774339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48802 (local address 127.22.225.189:37899)
0928 06:33:11.774379 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:11.774381 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:11.777994 (+ 3613us) server_negotiation.cc:407] Connection header received
0928 06:33:11.778111 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:11.778113 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:11.778126 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:11.778159 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:11.778581 (+ 422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48802: BlockingRecv error: recv got EOF from 127.0.0.1:48802 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 06:33:11.798053 13722 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:33:11.796919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48808 (local address 127.22.225.189:37899)
0928 06:33:11.796968 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:11.796970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:11.797366 (+ 396us) server_negotiation.cc:407] Connection header received
0928 06:33:11.797485 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:11.797487 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:11.797501 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:11.797533 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:11.797990 (+ 457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48808: BlockingRecv error: recv got EOF from 127.0.0.1:48808 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 06:33:11.817343 13722 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:33:11.808926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48822 (local address 127.22.225.189:37899)
0928 06:33:11.808981 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:11.808983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:11.816641 (+ 7658us) server_negotiation.cc:407] Connection header received
0928 06:33:11.816764 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:11.816765 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:11.816778 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:11.816810 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:11.817280 (+ 470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48822: BlockingRecv error: recv got EOF from 127.0.0.1:48822 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260928 06:33:11.836725 13722 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:33:11.832907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48830 (local address 127.22.225.189:37899)
0928 06:33:11.832957 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:11.832960 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:11.835909 (+ 2949us) server_negotiation.cc:407] Connection header received
0928 06:33:11.836085 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:11.836086 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:11.836099 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:11.836150 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:11.836657 (+ 507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48830: BlockingRecv error: recv got EOF from 127.0.0.1:48830 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260928 06:33:11.855348 13722 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:33:11.854368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48834 (local address 127.22.225.189:37899)
0928 06:33:11.854420 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:11.854422 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:11.854719 (+ 297us) server_negotiation.cc:407] Connection header received
0928 06:33:11.854827 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:11.854828 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:11.854842 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:11.854872 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:11.855286 (+ 414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48834: BlockingRecv error: recv got EOF from 127.0.0.1:48834 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260928 06:33:11.889869 13722 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:33:11.889031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48842 (local address 127.22.225.189:37899)
0928 06:33:11.889082 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:11.889085 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:11.889258 (+ 173us) server_negotiation.cc:407] Connection header received
0928 06:33:11.889324 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:11.889326 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:11.889337 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:11.889422 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:11.889795 (+ 373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48842: BlockingRecv error: recv got EOF from 127.0.0.1:48842 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 06:33:11.955845 13722 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:33:11.955136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48846 (local address 127.22.225.189:37899)
0928 06:33:11.955196 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:11.955198 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:11.955299 (+ 101us) server_negotiation.cc:407] Connection header received
0928 06:33:11.955364 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:11.955365 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:11.955378 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:11.955438 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:11.955783 (+ 345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48846: BlockingRecv error: recv got EOF from 127.0.0.1:48846 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260928 06:33:12.085961 13722 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:33:12.085071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48856 (local address 127.22.225.189:37899)
0928 06:33:12.085144 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:12.085146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:12.085291 (+ 145us) server_negotiation.cc:407] Connection header received
0928 06:33:12.085352 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:12.085353 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:12.085369 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:12.085456 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:12.085893 (+ 437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48856: BlockingRecv error: recv got EOF from 127.0.0.1:48856 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260928 06:33:12.343935 13722 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:33:12.343207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48862 (local address 127.22.225.189:37899)
0928 06:33:12.343290 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:12.343292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:12.343320 (+ 28us) server_negotiation.cc:407] Connection header received
0928 06:33:12.343357 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:12.343358 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:12.343369 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:12.343433 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:12.343844 (+ 411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48862: BlockingRecv error: recv got EOF from 127.0.0.1:48862 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 06:33:12.638404 13830 catalog_manager.cc:1259] Loaded cluster ID: a5f0f723a62b4d25a3ba471f73f0997f
I20260928 06:33:12.638513 13830 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f5103cadac8e4f61b924dda3a102d319: loading cluster ID for follower catalog manager: success
I20260928 06:33:12.639348 13830 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f5103cadac8e4f61b924dda3a102d319: acquiring CA information for follower catalog manager: success
I20260928 06:33:12.639686 13830 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f5103cadac8e4f61b924dda3a102d319: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 06:33:12.694881 13857 catalog_manager.cc:1259] Loaded cluster ID: a5f0f723a62b4d25a3ba471f73f0997f
I20260928 06:33:12.694962 13857 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 57be79ee1ec248d8845319b34f696031: loading cluster ID for follower catalog manager: success
I20260928 06:33:12.695812 13857 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 57be79ee1ec248d8845319b34f696031: acquiring CA information for follower catalog manager: success
I20260928 06:33:12.696154 13857 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 57be79ee1ec248d8845319b34f696031: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 06:33:12.860255 13931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:12.857167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48868 (local address 127.22.225.189:37899)
0928 06:33:12.857286 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:12.857288 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:12.857332 (+ 44us) server_negotiation.cc:407] Connection header received
0928 06:33:12.857433 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:12.857434 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:12.857490 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:12.857583 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:33:12.857763 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:12.858256 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:12.859422 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:12.859633 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:12.859701 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:12.859886 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:33:12.859998 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:33:12.860062 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:12.860128 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 06:33:12.860149 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":69,"threads_started":1}
I20260928 06:33:12.864980 13937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:12.862176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44030 (local address 127.22.225.188:34885)
0928 06:33:12.862346 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:12.862348 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:12.862358 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:33:12.862405 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:12.862406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:12.862460 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:12.862565 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:33:12.862650 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:12.863382 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:12.864100 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:12.864295 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:12.864353 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:12.864556 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:33:12.864774 (+ 218us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:33:12.864851 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:12.864894 (+ 43us) server_negotiation.cc:299] Negotiation successful
0928 06:33:12.864913 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":77,"threads_started":1}
I20260928 06:33:12.869762 13931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:12.866884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48878 (local address 127.22.225.189:37899)
0928 06:33:12.866930 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:12.866932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:12.867096 (+ 164us) server_negotiation.cc:407] Connection header received
0928 06:33:12.867156 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:12.867157 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:12.867167 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:12.867222 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:33:12.867394 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:12.867872 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:12.869014 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:12.869224 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:12.869257 (+ 33us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:12.869471 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:33:12.869512 (+ 41us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:33:12.869565 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:12.869648 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 06:33:12.869671 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 06:33:12.869860 13937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:12.866764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44032 (local address 127.22.225.188:34885)
0928 06:33:12.866806 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:12.866808 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:12.866998 (+ 190us) server_negotiation.cc:407] Connection header received
0928 06:33:12.867024 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:12.867025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:12.867036 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:12.867090 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:33:12.867249 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:12.867960 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:12.869002 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:12.869297 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:12.869360 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:12.869514 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:33:12.869572 (+ 58us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:33:12.869622 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:12.869757 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 06:33:12.869784 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:33:12.870481 13944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:12.866710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54956 (local address 127.22.225.190:46789)
0928 06:33:12.866829 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:12.866830 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:12.866897 (+ 67us) server_negotiation.cc:407] Connection header received
0928 06:33:12.867026 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:12.867028 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:12.867084 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:12.867163 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:33:12.867339 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:12.868117 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:12.868923 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:12.869217 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:12.869332 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:12.869572 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:33:12.869679 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:33:12.870097 (+ 418us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:12.870360 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 06:33:12.870384 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":61,"threads_started":1}
I20260928 06:33:12.878064 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 13575
I20260928 06:33:12.885362 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 13650
I20260928 06:33:12.892910 23430 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcCVV2M/build/release/bin/kudu with pid 13725
2026-09-28T06:33:12Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2936 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6131 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (293756 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST